Syncing rgb and depth timestamps

WouterOddBot

Hi,

We have the rgb and mono camera running at the same framerate. Then we still need to sync the rgb and depth timestamps so that they're exactly the same. We use this script:

def create_sequence_number_sync_script(stream1_in: str, stream2_in: str, stream1_out: str) -> str:

return f"""

from math import isclose

stream1_in = node.io['{stream1_in}']

stream2_in = node.io['{stream2_in}']

stream1_out = node.io['{stream1_out}']

frame1 = None

frame2 = None

while True:

if frame1 is None:

frame1 = stream1_in.get()

if frame2 is None:

frame2 = stream2_in.get()

sequence_number1 = frame1.getSequenceNum()

sequence_number2 = frame2.getSequenceNum()

if sequence_number1 == sequence_number2:

timestamp1 = frame1.getTimestamp()

timestamp2 = frame2.getTimestamp()

assert isclose(timestamp1.total_seconds(), timestamp2.total_seconds(), rel_tol=0.001)

frame1.setTimestamp(timestamp2)

stream1_out.send(frame1)

frame1 = None

frame2 = None

elif sequence_number1 < sequence_number2: # if stream1 is behind ...

frame1 = None # ... dismiss stream1 frame

# ... and keep stream2 frame

else: # if stream2 is behind ...

# ... keep stream1 frame

frame2 = None # ... and dismiss stream2 frame

"""

In which stream1 is the rgb stream and stream2 the depth stream. The script replaces frame1's (rgb) timestamp with frame2's (depth) timestamp. This works well, except that every once in a while we get an rgb frame with a timestamp just 1 microsecond behind the depth timestamp, where I'd expect that they are exactly the same.

How can we prevent this? Is this due to some rounding error somewhere setting frame1's timestamp?

jakaskerl

WouterOddBot
If syncing is the main priority of the script, why not just use a sync node which has this functionality covered and is prone to user error?

Thanks,
Jaka

WouterOddBot

Hi jakaskerl

We've looked at the sync and demux nodes indeed. But afaik they don't give the synced frames identical timestamps. The depthai camera is just a small step in our entire Ros pipeline. For the Ros pipeline we need the rgb and depth timestamps to be identical.

Apart from that we already push the depthai camera to its computational limits. Given our setup where we can easily sync by sequence numbers the sync node based on timestamps in combination with the demux node seems quite a bit of overkill.

jakaskerl

Hi WouterOddBot

WouterOddBot But afaik they don't give the synced frames identical timestamps.

No amount of software syncing can achieve that.

If the frames were not taken at the same time, they will not have the same timestamp regardless of what you do. For perfect sync (almost perfect), you need HW syncing - to trigger the capture at exactly the same time - docs.

Thanks,
Jaka

WouterOddBot

jakaskerl This is not what I'm concerned about. I just provided the script for some context. It does exactly what we want, except this:

When I set a frame's timestamp with setTimestamp(timedelta) the frame is somehow not guaranteed to have exactly that timestamp. It's timestamp may be 1 microsecond off, probably due to some rounding error. That sounds like a bug in the depthai code.

You can reproduce this by copying one frame's timestamp to another frame (like the script above does) and then comparing their timestamps. You'd expect both frames to always have identical time frames, but every now and then there's a microsecond difference.

So I'd like to know if there's a way to fix this. I tried using setTimestampDevice(timedelta) but somehow that method is in the API but not accessible in the python library.

I hope this clarifies the problem a bit?

jakaskerl

WouterOddBot
Thanks, I'll forward this to dev team.

jakaskerl

Hi WouterOddBot
setTimestamp() on script node sets the device timestamp. Reading the timestamp on host via getTimestapDevice will work. getTimestamp will not since the value is not updated along with device. This is to be fixed.

Thanks,
Jaka

WouterOddBot

Hi @jakaskerl ,
Ok thanks for your explanation. We now compare frames using getTimestampDevice and it works.