Host clock sync

KyleSemelka

Hello,

We are using an Oak-D Pro W PoE on Ubuntu with DepthAI v2.24.0 and testing time sync reliability. What we've found is that the timestamp from dai::ImgFrame::getTimestamp is affected by Linux system clock jumps. Is this expected even though this timestamp is derived from steady_clock?

In our testing, after jumping the Linux system clock forward by 300 ms using the date -s command, it takes 40 seconds for the frame latency metric to stabilize.

We did the same test using dai::ImgFrame::getTimestampDevice and this timestamp is not affected by system clock jumps. However, since it's not synced with the host clock time, we cannot directly use this to calculate frame latency.

MatevzMorato

Hi @KyleSemelka !

Thanks for the report.
This is not expected, we will look into it and I'll reply here once we have more data.

MatevzMorato

We've double checked and we use std::steady_clock for syncing in all places.

Can you double check that std::steady_clock isn't causing issues already on your system?
It seems that it does sometimes happen: https://stackoverflow.com/questions/63054071/how-to-measure-elapsed-time-without-being-affected-by-changes-in-system-time.

KyleSemelka

MatevzMorato

I've double-checked the std::steady_clock behavior on my system. It is indeed not affected by jumps in the system clock. We are using Ubuntu 22.04 and Linux kernel 6.6.

Interestingly, we only see this behavior from the Oak-D when the host system clock jumps <450 ms. If the clock jumps >450 ms, then the received timestamps don't show any deviation.

KyleSemelka

I did a bit more digging. Some findings:

The timestamps from the Oak-D device periodically have huge jumps from one frame to the next (see "Timestamp rate of change" plot). Is this due to the time synchronization implementation?
We use getQueueEvents to wait for the next available frame. Plotting a moving average shows that after the time jump, we have lower execution time during that function call, and this execution time gradually reverts back to the normal value as the synchronization recovers.
The first-order difference between timestamps is almost exactly 100 ms, except for the large jumps.
The frame latency plot after the time jump shows a negative slope that gradually increases to 0. When I jump the system clock in the negative direction, this shows a positive slope that gradually decreases to 0.

Raw CSV data for each plot: https://drive.google.com/drive/folders/1604ymm_kulyncOfYhrPd4cKeG6MJZB5O?usp=sharing

aiyangsky

KyleSemelka

We are currently struggling with time synchronization, but our device is OAK-FFC4P.

In our tests, the OAK device had no way to synchronize the clock source with the host - we ran a 10Hz timer on the host and set the OAK camera to sample images at 10Hz, using the timestamps of the images to compare with the timestamps produced by the timer. We find that their differences accumulate. Therefore, it is determined that OAK is not time synchronized with the host.

Are you concerned about the above situation?

We hope that you can describe your test method in detail as an important reference for us, especially the meaning and access method of each data in your document.

Thank you in advance.

Sky