I'm trying to capture images of tree canopies with all three cameras on the Oak-D Lite at the same time. Due to the fine structure of the tree canopy, any delay between the depth map and color image can make it very difficult to associate structures in the two images. I'm currently syncing the streams with the sequence numbers, but am seeing \~150ms offset when starting the capture. I am relying on the firmware syncing to sync the camera triggers. But this can take more than 6 minutes to converge when the cameras are set to 10fps. [upl-image-preview url=https://discuss.luxonis.com/assets/files/2024-03-16/1710579056-793609-image.png] It is interesting to note that the camera's sync much faster once the error is less than the single frame time. But this means at higher framerates it takes much longer to get under the lower threshold. Is it possible to configure any parameters regarding the syncing of the cameras? For my application I don't mind discontinuities because I can't start capturing until they have converged. My understanding of this system is based on this section in the docs: [Sequence number syncing](https://docs.luxonis.com/projects/api/en/latest/tutorials/message_syncing/?highlight=software%20sync#sequence-number-syncing)

Thanks @"erik"#175, @"Luxonis-Alex"#68. Alex your work around should be good for my application. Here are my results for testing with a slight FPS offset on the RGB camera. [upl-image-preview url=https://discuss.luxonis.com/assets/files/2024-03-19/1710880157-9476-image.png] Im seeing a steady state error of 0.5ms for 0.00999 offset, 0.3ms for 0.00499 offset, and 0.05ms for no offset. These errors are no issue for us. Thank you for you help.

Is it possible to configure timestamp alignment between color and mono cameras?

JMcC

I'm trying to capture images of tree canopies with all three cameras on the Oak-D Lite at the same time. Due to the fine structure of the tree canopy, any delay between the depth map and color image can make it very difficult to associate structures in the two images. I'm currently syncing the streams with the sequence numbers, but am seeing ~150ms offset when starting the capture. I am relying on the firmware syncing to sync the camera triggers. But this can take more than 6 minutes to converge when the cameras are set to 10fps.

It is interesting to note that the camera's sync much faster once the error is less than the single frame time. But this means at higher framerates it takes much longer to get under the lower threshold.

Is it possible to configure any parameters regarding the syncing of the cameras? For my application I don't mind discontinuities because I can't start capturing until they have converged.

My understanding of this system is based on this section in the docs: Sequence number syncing

erik

Hi @JMcC ,
I believe we improved syncing (by adding line skipping) only between IMX378 and OV9282 (so non-lite cameras), and I can see this delay as well. I have added it to our roadmap, but it's not a high prio.

Luxonis-Alex

IMX214 (the color sensor of OAK-D-Lite) doesn't have a hardware frame sync feature, so we're relying on soft frame sync here as well. I think some timings changed in how the sensors are started during initialization, and it takes significantly longer for sync to converge, especially for the higher resolutions (4K+).

We need to improve this in FW, but for now a workaround can be to slightly reduce the color camera FPS:

camRgb.setFps(10)
camLeft.setFps(10)
camRight.setFps(10)

camRgb.setFps(camRgb.getFps()-0.00999)

^ the FW checks for FPS between all cams to be within 0.01 in order to enable soft-sync, so that's about the max difference that can be applied currently.

JMcC

Thanks @erik, @Luxonis-Alex.

Alex your work around should be good for my application. Here are my results for testing with a slight FPS offset on the RGB camera.

Im seeing a steady state error of 0.5ms for 0.00999 offset, 0.3ms for 0.00499 offset, and 0.05ms for no offset. These errors are no issue for us. Thank you for you help.

Luxonis-Alex

@JMcC Looks good!
I'm just a bit curious about what features/nodes your pipeline uses, and what camera resolutions are set. As on my side I'm seeing the initial time diff of about 125ms, and the time to converge is shorter.
I tested with depthai v2.25.0.0 and the simple pipeline here:
luxonis/depthai-experimentsblob/c867d79/gen2-syncing/host-frame-sync.py
slightly changed as:

@@ -40,9 +40,10 @@ for c in cam_list:
     xout[c].setStreamName(c)
     if c == 'rgb':
         cam[c] = pipeline.create(dai.node.ColorCamera)
-        cam[c].setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
-        cam[c].setIspScale(720, 1080)  # 1920x1080 -> 1280x720
-        cam[c].isp.link(xout[c].input)
+        cam[c].setResolution(dai.ColorCameraProperties.SensorResolution.THE_13_MP)
+        # Matching frame sizes needed for this script's frame stacking
+        cam[c].setVideoSize(640,480)
+        cam[c].video.link(xout[c].input)
     else:
         cam[c] = pipeline.create(dai.node.MonoCamera)
         cam[c].setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P)
@@ -50,6 +51,7 @@ for c in cam_list:
     cam[c].setBoardSocket(cam_socket_opts[c])
     cam[c].setFps(args.fps)
 
+cam['rgb'].setFps(cam['rgb'].getFps()-0.00999)

python3 host-frame-sync.py -f 10 -d -vv

   Seq  Left_tstamp  RGB-Left  Right-Left  Dropped
   num    [seconds]  diff[ms]    diff[ms]  delta
     0     0.168524  -125.131       0.011
     1     0.268512  -125.015       0.012
     2     0.368501  -124.900       0.011
     3     0.468490  -124.786       0.011
     4     0.568479  -124.671       0.011
     5     0.668468  -124.556       0.012
     6     0.768457  -124.442       0.012
     7     0.868446  -124.326       0.011
     8     0.968435  -124.212       0.012
     9     1.068424  -124.097       0.011
    10     1.168413  -123.983       0.011
...
   214    21.566212  -100.577       0.013
   215    21.666201  -100.463       0.012
   216    21.766190  -100.348       0.011
   217    21.866179  -100.233       0.012
   218    21.966170  -100.121       0.011
   219    22.066157  -100.003       0.015
   220    22.166146   -99.889       0.014
   221    22.266135   -99.775       0.014
   222    22.366124   -98.662       0.014
   223    22.466114   -98.548       0.016
   224    22.566102   -97.435       0.014
   225    22.666091   -97.320       0.015
   226    22.766080   -96.207       0.014
...
   377    37.864488    -4.035       0.012
   378    37.964477    -2.922       0.012
   379    38.064466    -2.807       0.011
   380    38.164455    -1.695       0.011
   381    38.264444    -1.580       0.012
   382    38.364433    -0.467       0.012
   383    38.464422    -0.353       0.010
   384    38.564411     0.551       0.014
   385    38.664400     0.666       0.014
   386    38.764389     0.507       0.015
   387    38.864378     0.621       0.014
   388    38.964368     0.483       0.017
   389    39.064356     0.599       0.014
   390    39.164345     0.472       0.014
   391    39.264334     0.587       0.014
...

Anyway we'll need to improve this in firmware. The cameras are configured and started in sequence, and other nodes initialization could add to the device CPU load and insert delays. For the start-streaming commands, we need a more tight control on timings.