Hello,

We are using an Oak-D Pro W PoE on Ubuntu with DepthAI v2.24.0 and testing time sync reliability. What we've found is that the timestamp from dai::ImgFrame::getTimestamp is affected by Linux system clock jumps. Is this expected even though this timestamp is derived from steady_clock?

In our testing, after jumping the Linux system clock forward by 300 ms using the date -s command, it takes 40 seconds for the frame latency metric to stabilize.

We did the same test using dai::ImgFrame::getTimestampDevice and this timestamp is not affected by system clock jumps. However, since it's not synced with the host clock time, we cannot directly use this to calculate frame latency.

Hi @KyleSemelka !

Thanks for the report.
This is not expected, we will look into it and I'll reply here once we have more data.

6 days later

MatevzMorato

I've double-checked the std::steady_clock behavior on my system. It is indeed not affected by jumps in the system clock. We are using Ubuntu 22.04 and Linux kernel 6.6.

Interestingly, we only see this behavior from the Oak-D when the host system clock jumps <450 ms. If the clock jumps >450 ms, then the received timestamps don't show any deviation.

    I did a bit more digging. Some findings:

    1. The timestamps from the Oak-D device periodically have huge jumps from one frame to the next (see "Timestamp rate of change" plot). Is this due to the time synchronization implementation?
    2. We use getQueueEvents to wait for the next available frame. Plotting a moving average shows that after the time jump, we have lower execution time during that function call, and this execution time gradually reverts back to the normal value as the synchronization recovers.
    3. The first-order difference between timestamps is almost exactly 100 ms, except for the large jumps.
    4. The frame latency plot after the time jump shows a negative slope that gradually increases to 0. When I jump the system clock in the negative direction, this shows a positive slope that gradually decreases to 0.

    Raw CSV data for each plot: https://drive.google.com/drive/folders/1604ymm_kulyncOfYhrPd4cKeG6MJZB5O?usp=sharing

      KyleSemelka

      Hi

      We are currently struggling with time synchronization, but our device is OAK-FFC4P.

      In our tests, the OAK device had no way to synchronize the clock source with the host - we ran a 10Hz timer on the host and set the OAK camera to sample images at 10Hz, using the timestamps of the images to compare with the timestamps produced by the timer. We find that their differences accumulate. Therefore, it is determined that OAK is not time synchronized with the host.

      Are you concerned about the above situation?

      We hope that you can describe your test method in detail as an important reference for us, especially the meaning and access method of each data in your document.

      Thank you in advance.

      Sky

        3 months later

        After some more debugging, this timesync behavior happens when the host clock is slewing. It seems that the Oak D time sync solution is not robust to host clock slewing -- at least at the rate that our system slews at. CLOCK_MONOTONIC (std::steady_clock on Linux) is affected by slewing due to NTP adjustments (adjtimex), so it makes sense that the time jump would still affect the time sync quality.