• DepthAI

  • Problem getting Stereo PostProcessing to work with DepthAlign

Hi,

I have an OAK-D Lite and want to output RGB-D Side by Side images for import into Looking Glass Holographic displays. I have combined multiple DepthAI examples includingr depth alignment to color camera, software time synchronization between depth and color, and postprocessing to filter the output for a smoother fewer gaps and higher quality depthmap.

However, I cannot get any of the post processing filters to work with Depth Alignment. As soon as I enable one filter I get this error at the first frame:

frame = msg.getFrame()

RuntimeError: ImgFrame doesn't have enough data to encode specified frame, required 4147200, actual 2073600. Maybe metadataOnly transfer was made?

I tried a lot of things, but the only think that works is disabling either postprocessing or depth alignment. The only other thing I tried that does not get this error is changing the setDepthAlign to DepthAlign.CENTER instead of the CameraBoardSocket.CAM_A. However, this results in the resolutions being different and not aligned. I will try to manually crop and scale the Depth map to the same as color and see if it is aligned.

Code is here: jimdinunzio/depthai-pythonblob/main/examples/StereoDepth/rgbd_camera.py

Any help would be appreciated.

Thanks.

    jimdinunzio
    Haven't tested this yet, but the required size is exactly 2x the actual. Consider linking depth instead of disparity. The size is UINT16 vs UINT8.

        jakaskerl Thanks, linking depth works. I needed to use actual depth anyway for the RGB-D.
        However, I sort of still want disparity for visualization. I tried to visualize actual depth and haven't found a good way. I also printed the type of the disp ImgFrame and it is always Type.RAW16
        property, but based on the docs below the actual data is different depending on disparity modes. I turned on subpixel mode which uses RAW16 internally, and the error is gone.
        This seems like a bug. For disparity modes encoding RAW8, the ImgFrame type should be RAW8 and not RAW16 and you would not be forced to use subpixel mode just to get it working. Am I missing something?

        From ImgFrame class Docs:
        depth

        Outputs ImgFrame message that carries RAW16 encoded (0..65535) depth data in depth units (millimeter by default). Non-determined / invalid depth values are set to 0

        disparity

        Outputs ImgFrame message that carries RAW8 / RAW16 encoded disparity data: RAW8 encoded (0..95) for standard mode; RAW8 encoded (0..190) for extended disparity mode; RAW16 encoded for subpixel disparity mode: - 0..760 for 3 fractional bits (by default) - 0..1520 for 4 fractional bits - 0..3040 for 5 fractional bits