Software time sync bug with multiple devices

Derek

Problem Statement

The Luxonis cameras (RGB or PoE) have “host clock syncing” — it automatically syncs device's timestamp to host's timestamp. Timestamp syncing happens continuously at around 5 second intervals. Device clocks are synced at below 500µs accuracy for PoE cameras, and below 200µs accuracy for USB cameras at 1σ (standard deviation) with host clock.

Each frame’s timestamp in host time can be stored to the host in a CSV. The intention was to use these timestamps (namely the first frame’s timestamp) to align and sync the videos from different camera streams. However testing and debugging has suggested that these host timestamps are not accurate / do not reflect the motions observed in the videos.

Software Version: depthai version 2.29.0

Hardware Version:

Video0: OAK-D-PRO W IMX378, SKU: A00573 https://shop.luxonis.com/products/oak-d-pro-w?variant=43715946447071
Video1: OAK-D-PRO Auto Focus, SKU A00546 https://shop.luxonis.com/products/oak-d-pro?variant=42455252369631

Luxonis docs on host clock sync

https://docs.luxonis.com/software/depthai-components/device/#Device-Clock-Host clock syncing

https://discuss.luxonis.com/blog/3270-host-clock-syncing-improved

https://docs.luxonis.com/hardware/platform/deploy/frame-sync/

The sync node does not currently support multiple device syncing, so if you want to sync messages from multiple devices, you should use the manual approach

Camera initialization code

class LuxonisCamera(AbstractCamera):
    def create_pipeline(self):
        color = self.pipeline.create(dai.node.ColorCamera)
        xoutGrp = self.pipeline.create(dai.node.XLinkOut)
        xoutGrp.setStreamName("xout")
        color.setResolution(dai.ColorCameraProperties.SensorResolution.THE_720_P)
        color.setCamera("color")
        color.setPreviewSize(1280, 720)
        color.setInterleaved(False)
        videoEnc = self.pipeline.create(dai.node.VideoEncoder)

        videoEnc.setDefaultProfilePreset(
            self.fps, dai.VideoEncoderProperties.Profile.H265_MAIN
        )
        videoEnc.setRateControlMode(dai.VideoEncoderProperties.RateControlMode.CBR)
        videoEnc.setBitrate(2000000)

        color.setColorOrder(dai.ColorCameraProperties.ColorOrder.RGB)
        color.setFps(self.fps)
        color.video.link(videoEnc.input)
        videoEnc.bitstream.link(xoutGrp.input)
        uvc = self.pipeline.createUVC()
        color.video.link(uvc.input)
        config = dai.Device.Config()
        config.board.uvc = dai.BoardConfig.UVC(1920, 1080)
        config.board.uvc.frameType = dai.ImgFrame.Type.NV12
        self.pipeline.setBoardConfig(config.board)

Camera initialization logs

2025-01-30T18:13:28.812402960Z INFO:libs.recorders.luxonis_recorder:Found 2 available devices.
2025-01-30T18:13:28.817118331Z INFO:libs.recorders.luxonis_recorder:Found device '1.3.1', MxId: '1844301051BA3AF500'
2025-01-30T18:13:28.822607147Z INFO:libs.recorders.luxonis_recorder:Found device '1.3.2', MxId: '18443010215F3EF500'
2025-01-30T18:13:32.149851218Z INFO:libs.cameras.luxonis_camera:Product name OAK-D-PRO-W mxid 18443010215F3EF500 with usb speed UsbSpeed.SUPER
2025-01-30T18:13:32.322623247Z INFO:libs.cameras.luxonis_camera:Product name OAK-D-PRO-AF mxid 1844301051BA3AF500 with usb speed UsbSpeed.SUPER
2025-01-30T18:13:32.323515183Z INFO:libs.recorders.luxonis_recorder:Successfully initialized 2 cameras
2025-01-30T18:13:32.323560805Z INFO:libs.recorders.luxonis_recorder:Starting Luxonis to record to /etc/standardbots/skills/echo-recording/2025-01-30/2025-01-30T18:13:28.294+00:00-Soda-Pouring-w-Luxonis-(1)
2025-01-30T18:13:32.323757234Z INFO:libs.cameras.luxonis_camera:Started recording OAK-D-PRO-AF with mxid 1844301051BA3AF500
2025-01-30T18:13:32.323990333Z INFO:libs.cameras.luxonis_camera:Starting H.265 recording to /etc/standardbots/skills/echo-recording/2025-01-30/2025-01-30T18:13:28.294+00:00-Soda-Pouring-w-Luxonis-(1)/video_1.h265 at 60 FPS on camera OAK-D-PRO-W with mxid 18443010215F3EF500
2025-01-30T18:13:32.324031829Z INFO:libs.cameras.luxonis_camera:Starting H.265 recording to /etc/standardbots/skills/echo-recording/2025-01-30/2025-01-30T18:13:28.294+00:00-Soda-Pouring-w-Luxonis-(1)/video_0.h265 at 60 FPS on camera OAK-D-PRO-AF with mxid 1844301051BA3AF500

Example video from Jan 30

The raw videos are clearly not synced, one video starts about 0.5 seconds earlier than the other (visible from motion of the waving hand)
However the host timestamps are nearly synced (0.06 seconds difference). This suggests the host timestamps are not accurate.

lux-video-0.mp4

3MB

lux-video-1.mp4

3MB

Screenshot from video

Analysis of host timestamps

From host clock syncing, cameras should be 0.06 seconds apart

0th frame comparison: 402.533 vs 402.597

Analysis of device timestamps

Visually, the recordings look closer to ~0.5 seconds apart. Anecdotally, this appears to be closer to the difference in device timestamps since device bootup (difference of 0.49 seconds)

Experiment: comparing visual time difference by recording a stopwatch

We used two cameras (Oak-D Pro and Oak-D Pro W) to record a stopwatch on an iPhone, and compared the 0th frame. From the frame-by-frame analysis the recordings are 0.63 seconds apart (38 frames @ 60 Hz). This is significantly larger than the difference in host timestamps for 0th frame reported by depthai.

The system timestamp is consistently 0.04 seconds (40 ms) ahead of the host timestamp logged by depthai
The difference in the host and system timestamps indicates the 0th frame of Video_0 and Video_1 is around 0.05-0.06 seconds (50-60 ms)
None of these deltas approach the order of magnitude in the visual analysis - 0.63 seconds (630 ms)

Code for logging host and system timestamps

h265 depthai video encoder code based on docs: https://docs.luxonis.com/software/depthai/examples/rgb_encoding/

with open(video_f, "wb") as videoFile, open(
    host_timestamps_f, "w"
) as host_ts_file, open(device_timestamps_f, "w") as dev_ts_file:
    host_ts_file.write("frame_number,t_s,t_us,sys_t_s,sys_t_us\\n")
    dev_ts_file.write("frame_number,t_s,t_us\\n")

    while self.is_recording:
        if h265Packet is not None:
	    # log system timestamp directly
            current_time = time.monotonic()
            sys_seconds = int(current_time)
            sys_microseconds = int((current_time - sys_seconds) * 1e6)

	    # get the h265 packet from depthai
            h265Packet.getData().tofile(videoFile)
            # log host timestamp from depthai
            host_ts = h265Packet.getTimestamp()
            device_ts = h265Packet.getTimestampDevice()
            host_total_seconds = int(host_ts.total_seconds())
            host_microseconds = host_ts.microseconds
            dev_total_seconds = int(device_ts.total_seconds())
            dev_microseconds = device_ts.microseconds
            host_ts_file.write(
                f"{self.frames_written},{host_total_seconds},{host_microseconds},{sys_seconds},{sys_microseconds}\\n"
            )
            dev_ts_file.write(
                f"{self.frames_written},{dev_total_seconds},{dev_microseconds}\\n"
            )
            if self.host_monotonic_time_start is None:
                self.host_monotonic_time_start = float(
                    host_total_seconds
                ) + (host_microseconds / 1e6)
            self.frames_written += 1

jakaskerl

Derek The sync node does not currently support multiple device syncing, so if you want to sync messages from multiple devices, you should use the manual approach

Where are you attempting to sync the devices together? Do keep in mind that since the triggering is not set via hardware, there will always be some difference (max 1/FPS*2).

Thanks,
Jaka

Derek

jakaskerl

Hi Jaka, it looks like the device timestamp is not automatically syncing to the host timestamp as I would expect. I did a ground-truth test, where I pointed three cameras at a stopwatch and started recording frames with the initialization and h265 recording code I shared above. There was a large delay between video0 and video1's first frame (0.55 seconds) that is not reflected in the host clocks reported.

Please take a look at these screenshots:

Video_0: 10.33 stopwatch seconds in the 0th frame (ground truth), 12002.854 host clock, 6.729 device clock

Video_1: 10.88 stopwatch seconds in the 0th frame (ground truth), 12002.864 host clock, 6.219 device clock

Video_3: 10.32 stopwatch seconds in the 0th frame (ground truth), 12002.897 host clock, 5.131 device clock

The host clock timestamps for all videos are within 43 ms. This does not match the ground truth, where video_0 and video_1 have a difference of 550 ms.

Software Version: depthai version 2.29.0

Hardware Version:

Video_0: OAK-D-PRO W IMX378, SKU: A00573 https://shop.luxonis.com/products/oak-d-pro-w?variant=43715946447071
Video_1: OAK-D-PRO Auto Focus, SKU A00546 https://shop.luxonis.com/products/oak-d-pro?variant=42455252369631
Video_3: OAK-D Pro PoE