Hi,

I found code to align and sync RGB+D data (using the API), and code to record camera streams (using the SDK). Is there a script that does all three? I haven't found a way to merge it all.

I am using the OAK-D S2 camera and am interested in recording RGB and depth data in an .mp4 format, resolutions of 720p at 30fps. It would also be nice if the code incorporates recording stereo vision outputs (left and right).

Thank you very much.

    Hi gloria
    RGBD can be recorded to an mcap. If you wish to record an mp4, you will need to create two separate recordings, one for RGB, the other one for disparity. Both can be encoded using the videoencoder, but to have them overlapping you would need custom logic on host side.

    Thoughts?
    Jaka

      Hi Jaka,

      Thank you for your response. I would like to confirm that you are referring to the following code for MCAP recording? https://github.com/luxonis/depthai/blob/main/depthai_sdk/examples/recording/mcap_record.py

      I've been using the mp4 recording method (code below) but find that the RGB FOV is a zoomed-in version of the depth data. For instance, if I take a video of myself, I can only see part of my head/face in the RGB video but I can see the whole thing and more behind me in the depth video. Am I missing something? This is why I ended up resorting to the RGB-D alignment code.

      Thank you so much for your help.

        Hi gloria
        To align the depth to color camera you need to specify the stereoconfig:
        stereo.config_stereo(align=color)
        I also think 1080p is not supported on mono cameras, so they are set to 720p. You will need to downscale the color camera (ispScale 2/3).

        If you wish to blend the streams together, i suggest you create a callback function for the visualizer in which you use the code from API version of RGB_depth_align.

        Thanks,
        Jaka

          Hi gloria
          "depth" variable in your code is a stereo component. You can call depth.config_stereo(align=color) (sorry, I was using the same naming scheme as in the stock example for mcap-recording)

          Hope this helps,
          Jaka

            Hi Jaka,

            Thank you so much for all of your help. If I may bother you with one last question (I am hoping this is the last issue I need to resolve). Please see below the error message I am receiving, I am not sure what might be wrong with the code.

            Thanks

              Hi gloria
              Update the AV package, depthai respository (git pull) and depthai_sdk package. Then change the out.depth to out.disparity. It should work, I just tested it myself.

              Thanks,
              Jaka

                Hi Jaka,

                Please see below, it seems like it has an issue with align=color.

                From my understanding, disparity is not the same as depth. Shouldn't I be using out.depth instead?

                Thanks

                  Hi gloria
                  Could you add your code as well please?

                  Disparity is similar to depth in a sense that depth is calculated from disparity. The advantage disparity has over depth is that it can be encoded using VideoEncoder.

                  Thanks,
                  Jaka

                    Hi gloria
                    Add this to the top

                    import depthai
import av
import depthai_sdk
print('depthai module: ', depthai.__version__)
print('depthai_sdk module: ', depthai_sdk.__version__)
print('av module: ', av.__version__)

                    I am running the exact same code and I am getting no error.

                    Thanks,
                    Jaka

                      Hi Jaka,

                      I would like to thank you for your patience and all of your help. It is working for me now! I do have one last question about how I can get the depth from the disparity? I can't seem to find it in your documentation.

                      Warm regards,

                      Gloria

                      Is there no way to directly record depth as an .mp4 using this method?

                        Hi gloria
                        Guide here.

                        There is no way to record depth with videoencoder on device. You still can record depth, but you have to do it host side. You will have to find/code a way to save UINT16 depth to mp4. I'd suggest GPT for aid.

                        Thanks,
                        Jaka

                          2 years later

                          Thanks all in advance for the help. I recorded video with RGB, left and right camera with 20fps. The RGB was collected image with 12 MP and scale it to 1/4 (tried to have maximum FOV) and left/right has 720 P. Is there any solution to align the depth information using disprity map using left+right camera with RGB?

                          Thanks

                            12 days later