I would like to record a mono/rgb video and a stereo depth video at the same time, so that I can run detection and depth algorithms on the videos later. I am trying to record using this "Stereo Recording" example in the SDK https://docs.luxonis.com/projects/sdk/en/latest/samples/recording/SDK_stereo_record/?highlight=record. 1. However, when I run this code, no video footage is saved. Could someone help me out here? 2. Also, how would I record both mono/rgb and stereo depth simultaneously? Seems like it should be something like this: oak.visualize(stereo.out.disparity, color.out.main, record_path='disparity.mp4'). But I don't know how I would specify the record path of the color video. Lastly, if there is a better option for recording that you would recommend, please feel free to let me know! Thank you, Jae

mono/rgb + stereo depth recording (simultaneously)

jsiic

I would like to record a mono/rgb video and a stereo depth video at the same time, so that I can run detection and depth algorithms on the videos later.

I am trying to record using this "Stereo Recording" example in the SDK https://docs.luxonis.com/projects/sdk/en/latest/samples/recording/SDK_stereo_record/?highlight=record.

However, when I run this code, no video footage is saved. Could someone help me out here?
Also, how would I record both mono/rgb and stereo depth simultaneously? Seems like it should be something like this:
oak.visualize(stereo.out.disparity, color.out.main, record_path='disparity.mp4').
But I don't know how I would specify the record path of the color video.

Lastly, if there is a better option for recording that you would recommend, please feel free to let me know!

Thank you,
Jae

jakaskerl

Hi jsiic
Like it's done in https://docs.luxonis.com/projects/sdk/en/latest/samples/recording/SDK_mcap_record/, but change the recording type from MCAP to VIDEO.

jsiic Lastly, if there is a better option for recording that you would recommend, please feel free to let me know!

Depthai v2.26.0.0 should have record/replay features built in. It is supposed to release in approximately a month.

Thanks,
Jaka

jsiic

jakaskerl Hi jaka, I gave it a shot and got the following error:
any clue what's going on?

C:\...depthai_sdk\examples\recording\mcap_record.py 
[2024-01-24 17:47:22] INFO [root.__init__:118] Setting IR laser dot projector brightness to 800mA
mjpeg CAM_A_bitstream 1
[2024-01-24 17:47:22] WARNING [root.update:60] Exception while creating AvWriter: No module named 'av'.
Falling back to FileWriter, saving uncontainerized encoded streams.
[2024-01-24 17:47:22] INFO [root.close:456] Closing OAK camera
Traceback (most recent call last):
  File "C:\Users\kimjs\PycharmProjects\OakD\venv\Lib\site-packages\depthai_sdk\recorders\video_recorder.py", line 56, in update
    from .video_writers.av_writer import AvWriter
  File "C:\Users\kimjs\PycharmProjects\OakD\venv\Lib\site-packages\depthai_sdk\recorders\video_writers\__init__.py", line 4, in <module>
    from .video_writer import VideoWriter
  File "C:\Users\kimjs\PycharmProjects\OakD\venv\Lib\site-packages\depthai_sdk\recorders\video_writers\video_writer.py", line 5, in <module>
    import av
ModuleNotFoundError: No module named 'av'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\kimjs\PycharmProjects\OakD\depthai\depthai_sdk\examples\recording\mcap_record.py", line 14, in <module>
    oak.start(blocking=True)
  File "C:\Users\kimjs\PycharmProjects\OakD\venv\Lib\site-packages\depthai_sdk\oak_camera.py", line 487, in start
    handler.setup(self.pipeline, self.device, self._new_msg_callbacks)
  File "C:\Users\kimjs\PycharmProjects\OakD\venv\Lib\site-packages\depthai_sdk\classes\packet_handlers.py", line 191, in setup
    self.recorder.start(device, xouts)
  File "C:\Users\kimjs\PycharmProjects\OakD\venv\Lib\site-packages\depthai_sdk\record.py", line 106, in start
    self.recorder.update(self.path, device, xouts)
  File "C:\Users\kimjs\PycharmProjects\OakD\venv\Lib\site-packages\depthai_sdk\recorders\video_recorder.py", line 63, in update
    self._writers[xout_name] = FileWriter(self.path, file_name, fourcc)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\kimjs\PycharmProjects\OakD\venv\Lib\site-packages\depthai_sdk\recorders\video_writers\file_writer.py", line 15, in __init__
    super().__init__()
TypeError: BaseWriter.__init__() missing 2 required positional arguments: 'path' and 'name'
Sentry is attempting to send 2 pending error messages
Waiting up to 2 seconds
Press Ctrl-Break to quit

Process finished with exit code 1

jakaskerl

Hi jsiic
Looks like you are missing the av module. It's the python binding for ffmpeg. pip3 install av should work I think.

Thanks,
Jaka

jsiic

jakaskerl
Thanks, the recording is working properly. I tested it out the SDK Yolo-6 spatial detection with the video footage (color, left, and right) and it seems like the spatial coordinates are completely off. Even though I was moving very far away from the camera, the z value just stayed at 1.5m. When I changed the code to run off of live footage, it would accurately detect the change in depth.

Maybe it has to do with the resolution of the left/right footages, or the way I named the video files? Any ideas?

my code is

from depthai_sdk import OakCamera
import depthai as dai

with OakCamera(replay='C:/..../2-19443010D147741300') as oak:
    color = oak.create_camera('color')

    # List of models that are supported out-of-the-box by the SDK:
    # https://docs.luxonis.com/projects/sdk/en/latest/features/ai_models/#sdk-supported-models
    nn = oak.create_nn('yolov6nr3_coco_640x352', color, spatial=True)

    nn.config_spatial(
        bb_scale_factor=0.5, # Scaling bounding box before averaging the depth in that ROI
        lower_threshold=300, # Discard depth points below 30cm
        upper_threshold=10000, # Discard depth pints above 10m
        # Average depth points before calculating X and Y spatial coordinates:
        calc_algo=dai.SpatialLocationCalculatorAlgorithm.AVERAGE
    )


    oak.visualize([nn.out.main], fps=True)
    #oak.visualize(nn.out.passthrough)
    oak.start(blocking=True)

In the folder, I have the files: calib.json, color.mp4, left.mp4, right.mp4

jsiic

jakaskerl Nevermind, it works fine when I use mono footage with spatial detection. I think the issue was that the color footage had a different resolution from the left/right footages. Thanks!

jakaskerl

Hi @jsiic
What script did you use to record the files?
I tested with this and it worked:

from depthai_sdk import OakCamera, RecordType

with OakCamera() as oak:
    color = oak.create_camera('color', resolution='720p', fps=30)
    #color.config_color_camera(isp_scale=(2, 3)) # 720P
    left = oak.create_camera('left', resolution='400p', fps=30)
    right = oak.create_camera('right', resolution='400p', fps=30)

    # Sync & save all streams
    recorder = oak.record([color, left, right], './', RecordType.VIDEO)
    # recorder.config_mcap(pointcloud=True)
    oak.visualize(left)
    oak.start(blocking=True)

Thanks,
Jaka