Getting synced frames based on external trigger Oakd-S2

toon

Dear support,

I am trying to trigger my oakd-s2 poe with an external trigger, using depthai V3 and python 3.10. The idea is to to do software sync on the device to prevent unnecessary data transfer, while being able to trigger the camera's and get the synced left,right and rgb frames.

I used the following code as example sync.py and rewritten it to include also the RGB sensor. The code below is working properly with the RGB sensor set to false (rgb_enabled=False). By enabling the RGB camera, the idea was to stream the RGB sensor on the device, and by syncing one of the ir-sensors to only get all frames (left, right, and rgb) if the ir-camera is triggered. Somehow the code does not work when enabling the rgb camera. However if I disable the hardware_trigger (hardware_trigger=False) it is working fine, but of course streaming instead of sending only at at trigger . Any idea why this doesn't work?

import depthai as dai

import datetime

import cv2

rgb_enabled=True

hardware_trigger=True

fps=5

pipeline = dai.Pipeline()

if rgb_enabled:

rgb = pipeline.create(dai.node.Camera).build(dai.CameraBoardSocket.CAM_A, sensorFps=fps)

left = pipeline.create(dai.node.Camera).build(dai.CameraBoardSocket.CAM_B)

if hardware_trigger:

left.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)

left.initialControl.setExternalTrigger(4,3) #somehow 1,0 blocks after some time

# left.initialControl.setManualExposure(800, 200)

right = pipeline.create(dai.node.Camera).build(dai.CameraBoardSocket.CAM_C)

# right.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)

# right.initialControl.setExternalTrigger(4,3)

# right.initialControl.setManualExposure(800, 200)

sync = pipeline.create(dai.node.Sync)

sync.setRunOnHost(False) # Can also run on device

sync.setSyncThreshold(datetime.timedelta(milliseconds=100))

left.requestFullResolutionOutput().link(sync.inputs["left"])

right.requestFullResolutionOutput().link(sync.inputs["right"])

if rgb_enabled:

rgb.requestFullResolutionOutput().link(sync.inputs["rgb"])

outQueue = sync.out.createOutputQueue()

pipeline.start()

print("pipeline started, waiting on frames")

while pipeline.isRunning():

messageGroup : dai.MessageGroup = outQueue.get()

left = messageGroup["left"]

right = messageGroup["right"]

if rgb_enabled:

rgb = messageGroup["rgb"]

print(f"Timestamps, message group {messageGroup.getTimestamp()}, left-right {abs(right.getTimestamp()-left.getTimestamp())}, rgb-right {abs(rgb.getTimestamp()-right.getTimestamp())}")

cv2.imshow("rgb", rgb.getCvFrame())

else:

print(f"Timestamps, message group {messageGroup.getTimestamp()}, left-right {abs(right.getTimestamp()-left.getTimestamp())}")

cv2.imshow("left", left.getCvFrame())

cv2.imshow("right", right.getCvFrame())

if cv2.waitKey(1) == ord('q'):

break

toon

@bartvm

bartvm

Update:

After some debugging we found out that the syncThreshold should be set to 8000 ms, then the code works.
However, 8000 ms is a bit too much. We already downscaled the RGB resolution to (1920,1080), but difference between RGB and IR frames is still large ±4 seconds. Any ideas how to improve the code got get an acceptable time difference <200ms?

toon

In addition: If we run with hardware_tigger=False the timediff of the rgb-sensor compared to ir-sensor is around 0-50ms, as expected. However if we sethardware_trigger=True the timediff of the rgb stream ranges in the range of 3000-7000ms, while we did not adapt any settings of the rgb sensor. How is the rgb sensor affected by the settings of the ir sensor?

jakaskerl

toon
It is not, but if you keep the framesyncmode of RGB to OUTPUT (default) then camera will begin streaming at random. Make sure they are all INPUT. The difference should be sub-ms. Also keep left.initialControl.setExternalTrigger(4,3) #somehow 1,0 blocks after some time to 1, 0. Or something much lower than 4,3.

Thanks,
Jaka