Hardware trigger synchronization including RGB image (Oakd-s2)

jakaskerl sorry for the misconfusion, I am using a OAK-D S2 PoE with autofocus, so I have already implemented the M8 connector.

What I would like to do is to trigger multiple devices (3) with a external trigger pulse. Furthermore I want to output an rgb image as close as possible with the triggered mono-images, but they don't have to be fully hardware synchronised.

Dear Jakaskerl,

Indeed the color camera in following script does not work example.

Instead I wrote the following script including sync node, but still I do not get an image, what am I doing wrong?

#!/usr/bin/env python3
import depthai as dai
import cv2
import time
from datetime import timedelta

# FPS_tof = 10
FPS_rgb = 10
with_trigger = True


pipeline = dai.Pipeline()

# RGB camera
camRgb = pipeline.create(dai.node.ColorCamera)
camRgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.RGB)
camRgb.setIspScale(2, 3)
camRgb.setBoardSocket(dai.CameraBoardSocket.RGB)
camRgb.setFps(FPS_rgb)
if with_trigger:
    camRgb.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)
    camRgb.initialControl.setExternalTrigger(4, 3)


xout = pipeline.create(dai.node.XLinkOut)
xout.setStreamName("out")
# camRgb.isp.link(xout.input)  # Link the RGB output to XLinkOut

# Mono left camera
monoLeft = pipeline.create(dai.node.MonoCamera)
monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P)
monoLeft.setBoardSocket(dai.CameraBoardSocket.LEFT)
monoLeft.setFps(FPS_rgb)
if with_trigger:
    monoLeft.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)
    monoLeft.initialControl.setExternalTrigger(4, 3)

# Mono right camera
monoRight = pipeline.create(dai.node.MonoCamera)
monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P)
monoRight.setBoardSocket(dai.CameraBoardSocket.RIGHT)
monoRight.setFps(FPS_rgb)
if with_trigger:
    monoRight.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)
    monoRight.initialControl.setExternalTrigger(4, 3)

xoutRight = pipeline.create(dai.node.XLinkOut)
xoutRight.setStreamName("right")
monoRight.out.link(xoutRight.input)

# Sync node (if you want to sync all cameras)
sync = pipeline.create(dai.node.Sync)
# sync.setSyncThreshold(50)  # Sync threshold in milliseconds
sync.setSyncThreshold(timedelta(milliseconds=1000))

# Linking RGB, left, and right to the sync node
camRgb.isp.link(sync.inputs["color"])  # Link RGB to sync
monoLeft.out.link(sync.inputs["left"])
monoRight.out.link(sync.inputs["right"])

# You can link the sync output to XLinkOut for all cameras
sync.out.link(xout.input)
# sync.out.link(xoutLeft.input)
# sync.out.link(xoutRight.input)

# Device connection
device_id = "169.254.1.103"
with dai.Device(pipeline, dai.DeviceInfo(device_id)) as device:

    queue = device.getOutputQueue("out", 8, False)
    queueright=device.getOutputQueue("right")

    print("Starting...", queue)
    while True:
        # if queueright["right"].has():
        #     test=queueright["right"].get().getCvFrame()
        #     print("x")
        messageGroup: dai.MessageGroup = queue.get()
        frameMono_right: dai.ImgFrame = messageGroup["right"]
        frameMono_left: dai.ImgFrame = messageGroup["left"]
        # frameDepth_tof: dai.ImgFrame = messageGroup["depth_tof"]
        frameRgb: dai.ImgFrame = messageGroup["color"]
        # Blend when both received
        if frameMono_right is not None:
            if frameRgb is not None:
                print(frameMono_right.getSequenceNum())
                print(frameRgb.getSequenceNum())
                # print(messageGroup["depth"].getTimestamp()-messageGroup["rgb"].getTimestamp())
                # print(messageGroup["rgb"].getSequenceNum())
                # print(messageGroup["depth"].getSequenceNum())
                monoRight_img = frameMono_right.getCvFrame()
                # # cv2.imshow("rgb", frameRgb)
                # # cv2.imshow("depth", colorizeDepth(frameDepth.getCvFrame()))
                test=frameRgb.getCvFrame()
                print("It is working")
            else:
                print("no depth frame")
        else:
            print("no rgb frame")

        key = cv2.waitKey(1)
        if key == ord("q"):
            break


cv2.destroyAllWindows()

The trigger signal is just a signal triggered by a pushbutton (for test and verification). Same setup as in externally trigger fsync. In the end we would like to trigger with a PLC based on the encoder value (every x cm).

Ok, so if I understand correctly, there is no way to hardware-trigger the stereo frames and then find the most close rgb-frame softwarematticaly (even if we leave it streaming at 30FPS)?

Dear @jakaskerl

We have tested the provided code with an configured external trigger on 6hz. The left and rgb camera are streaming. However after ±1 minutes a fraction of the color images is under-exposed. It looks like that the fraction of the image, has an older frame? Could you help me out. I am using the following example script:

https://docs.luxonis.com/hardware/platform/deploy/frame-sync/#Sensor%20FSYNC%20support

Thanks in advance.

First minute it works, then in steps of 5 you will see more and more part of the image becoming darker, until you get something like the image above. How accurate does the trigger need to be? I am currently using a revolution pi to create the signal which I believe is accurate, however of course it can be a few microseconds off.