OakCamera.record left stream has one extra frame

artur-ag

I'm having an issue with depthai_sdk.OakCamera.record(). The left stream always ends up with one extra frame compared to the right stream, and it causes the two to be out of sync. And then when I use OakCamera(replay='myrecord'), the depth map is naturally wrong because of the wrong sync. But if I manually remove that initial frame and reencode then video, then they're back in sync again.

Here's my recording code. I'm using the OAK-FFC-3P and two AR0234. I'm using depthai 2.25.1.0, depthai-sdk 1.13.2, and if it matters, ffmpeg 7.0 and av 12.0.0. OS is Windows 11. Am I doing something wrong?

import time
from depthai_sdk import OakCamera
from depthai_sdk.record import RecordType

with OakCamera() as oak:
    left = oak.camera('left,c', '800p', fps=30, encode='h264')
    right = oak.camera('right,c', '800p', fps=30, encode='h264')
    left.node.setIspScale(2, 3)
    right.node.setIspScale(2, 3)

    oak.record([left.out.encoded, right.out.encoded], path='./', record_type=RecordType.VIDEO)
    oak.start(blocking=False)
    time.sleep(5)

jakaskerl

Hi @artur-ag
Could you print the timestamps of frames from each camera?
luxonis/depthai-experimentsblob/master/gen2-syncing/host-frame-sync.py

Thanks,
Jaka

artur-ag

Thanks for the reply @jakaskerl. I had to adapt the script: I don't have a center camera, only left and right. Here's the output:

   Seq  Left_tstamp  RGB-Left  Right-Left  Dropped
   num    [seconds]  diff[ms]    diff[ms]  delta
     1     0.077347     0.000      33.346    1
     2     0.110680     0.000      33.344
     3     0.144012     0.000      33.343
     4     0.177344     0.000      33.344
     5     0.210676     0.000      33.343
     6     0.244008     0.000      33.343
     7     0.277340     0.000      33.344
     8     0.310672     0.000      33.343
     9     0.344004     0.000      33.344
[continues with delta 33.3 ms for over 1000 frames]

33 ms is one frame at 30 fps. That's consistent with the 1 frame delay I'm seeing during recording.

jakaskerl

Hi @artur-ag
Ah, makes sense then. Did you remove the syncing part by any chance? This should be fixed inside the host-frame-sync script.

artur-ag

This is all I changed. And I don't see any explicit syncing being done in the script...

21c21
< cam_list = ['left', 'rgb', 'right']
---
> cam_list = ['left', 'right']
48c48
<         cam[c].setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P)
---
>         cam[c].setResolution(dai.MonoCameraProperties.SensorResolution.THE_1200_P)
64c64
<     allowed_instances = [0, 1, 2]  # Center (0) & Left (1) & Right (2)
---
>     allowed_instances = [1, 2]  # Center (0) & Left (1) & Right (2)
135c135
<             if not(seqnum['rgb'] == seqnum['left'] == seqnum['right']):
---
>             if not(seqnum['left'] == seqnum['right']):
141c141
<                 rgb_left_diff   = (tstamp['rgb']   - tstamp['left']) * 1000
---
>                 rgb_left_diff = 0
144d143
<                                      or abs(rgb_left_diff)   > 0.15

jakaskerl

Hi @artur-ag
I mean this part: luxonis/depthai-experimentsblob/master/gen2-syncing/host-frame-sync.py#L63-L91

Thanks

artur-ag

jakaskerl Right, I changed allowed_instances = [0, 1, 2] to allowed_instances = [1, 2] because I don't have a camera 0. All the rest is the same. Is the default syncing assuming that a center camera exists?

jakaskerl

Hi @artur-ag
Add .configure_syncing(enable, threshold), to manually set the syncing on the SDK.

Like in this example.

Thanks
Jaka

artur-ag

Ah, I completely missed that, should be exactly what I need. Unfortunately, I get an exception when I use that. I tried with threshold 17, 50 and 500 just to be sure, and it's always the same error. When this happens, no stream files are saved.

non-existing PPS 0 referenced
non-existing PPS 0 referenced
decode_slice_header error
no frame!
Exception in thread Thread-6:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/mnt/ssd/collision-avoidance/luxonis/.venv/lib/python3.8/site-packages/sentry_sdk/integrations/threading.py", line 69, in run
    reraise(*_capture_exception())
  File "/mnt/ssd/collision-avoidance/luxonis/.venv/lib/python3.8/site-packages/sentry_sdk/_compat.py", line 60, in reraise
    raise value
  File "/mnt/ssd/collision-avoidance/luxonis/.venv/lib/python3.8/site-packages/sentry_sdk/integrations/threading.py", line 67, in run
    return old_run_func(self, *a, **kw)
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/mnt/ssd/collision-avoidance/luxonis/.venv/lib/python3.8/site-packages/depthai_sdk/record.py", line 27, in _run
    recorder.write(name, frames[name])
  File "/mnt/ssd/collision-avoidance/luxonis/.venv/lib/python3.8/site-packages/depthai_sdk/recorders/video_recorder.py", line 97, in write
    self._writers[name].write(frame)
  File "/mnt/ssd/collision-avoidance/luxonis/.venv/lib/python3.8/site-packages/depthai_sdk/recorders/video_writers/av_writer.py", line 157, in write
    shape = self.get_dimension(frame)
  File "/mnt/ssd/collision-avoidance/luxonis/.venv/lib/python3.8/site-packages/depthai_sdk/recorders/video_writers/av_writer.py", line 90, in get_dimension
    frames = self._codec.decode(enc_packets[-1])
  File "av/codec/context.pyx", line 492, in av.codec.context.CodecContext.decode
  File "av/codec/context.pyx", line 507, in av.codec.context.CodecContext.decode
  File "av/codec/context.pyx", line 404, in av.codec.context.CodecContext._send_packet_and_recv
  File "av/error.pyx", line 328, in av.error.err_check
av.error.InvalidDataError: [Errno 1094995529] Invalid data found when processing input; last error log: [h264] no frame!

I suspect it's because the very first frame pair only has 1 valid frame. But I monkey-patched the function that throws an error, and now it works and both streams have exactly the same number of frames.

# .../depthai_sdk/recorders/video_writers/av_writer.py
def get_dimension(self, img: dai.ImgFrame) -> Optional[Tuple[int, int]]:
        enc_packets = self._codec.parse(img.getData())
        if len(enc_packets) == 0:
            return None
        try:
            frames = self._codec.decode(enc_packets[-1])  # this throws the error
        except:
            return (1280, 800)  # I hardcoded the size I'm using
        if not frames:
            return None
        return frames[0].width, frames[0].height

This is not a robust solution though...

jakaskerl

Hi @artur-ag
Did you try with any other version (1.13.1 is kinda broken sometimes..). But seems like SDK issue, yes.

Thanks,
Jaka

artur-ag

1.12.x doesn't have configure_syncing, and 1.13.1 has the same issue as 1.13.2. I guess that's just it. For my use-case, it's enough for me to monkey-patch av_writer.py, or remove the extra frame after the fact. Thank you for the help!
EDIT: For reference, PyPI has 1.13.2, but depthai_sdk.__version__ still reports 1.13.1. I tried both.

jakaskerl

artur-ag
Thanks for the heads-up.

SDK is currently deprecated to make room for new depthai which will have all this functionality integrated in C++.

Thanks,
Jaka