Artifacts in depth image

joshf

Hi all,
We're seeing major artifacts in the depth images from our OAK-D Pro POE, along edges in the scene -- has anyone seen something similar and/or know how to get rid of them? We've tried messing with all the settings we can think of and nothing has seemed relevant.

OAK-D point cloud:

Actual scene:

erik

Hi joshf ,
It's a bit hard to understand the perception, which artifacts are you referring to? At first, it looks like subpixel mode would remove the "layering", and perhaps a brightness filter for the edged of depth frame (which has invalid depth due to rectification). Also could you share rectified images? What's the power output on dot projector?
Thanks, Erik

joshf

Hi erik ,

I appreciate the response. The artifacts I'm referring to are at the edges of the cardboard box and between the rows of cereal boxes, especially in this area:

Would subpixel mode or brightness filtering help, in your opinion?

We've tried changing brightness from max to min and various intermediate values, without any noticeable change. The full camera settings we used for these images are as follows:

        monoResolution = dai.MonoCameraProperties.SensorResolution.THE_400_P

        left = self.pipeline.create(dai.node.MonoCamera)
        left.setResolution(monoResolution)
        left.setBoardSocket(dai.CameraBoardSocket.LEFT)
        left.setFps(fps)

        right = self.pipeline.create(dai.node.MonoCamera)
        right.setResolution(monoResolution)
        right.setBoardSocket(dai.CameraBoardSocket.RIGHT)
        right.setFps(fps)

        stereo = self.pipeline.create(dai.node.StereoDepth)
        stereo.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.HIGH_DENSITY)
        stereo.setPostProcessingHardwareResources(
            9, 6
        ) 

        stereo.setExtendedDisparity(True) 
        stereo.initialConfig.setMedianFilter(
            dai.StereoDepthProperties.MedianFilter.KERNEL_7x7  # not technically compatible with extended disparity but hasn't made a big difference
        )
        stereo.initialConfig.setConfidenceThreshold(200) 
        stereo.initialConfig.setDepthUnit(
            dai.RawStereoDepthConfig.AlgorithmControl.DepthUnit.MILLIMETER
        )

        left.out.link(stereo.left)
        right.out.link(stereo.right)

        config = stereo.initialConfig.get()

        config.postProcessing.speckleFilter.enable = True
        config.postProcessing.speckleFilter.speckleRange = 100

        config.postProcessing.temporalFilter.enable = True

        config.postProcessing.spatialFilter.enable = True
        config.postProcessing.spatialFilter.holeFillingRadius = 5
        config.postProcessing.spatialFilter.numIterations = 1

        config.postProcessing.thresholdFilter.minRange = int(
           .15 * 1000
        )  
        config.postProcessing.thresholdFilter.maxRange = int(
            1.5 * 1000
        )

        ...
        device = dai.Device(ip.value)
        # device.setIrLaserDotProjectorBrightness(765)

joshf

Rectified images from depthai_demo:

joshf

These are with subpixel on and max dot brightness: the artifacts have less layering, but they are still very much present:

erik

Hi joshf ,
I would first try the brightness filter, and enabling dot projector - it looks like it's disabled, and that texture isn't that great. In some parts it's overexposed, in some parts there isn't enough texture.
Could you also share the pointcloud itself?
Thanks, Erik

joshf

Hi erik ,
Where do i find the brightness filter? I tried searching the docs and didn't come across anything.
Is there a particular format which would be convenient for me to upload the point cloud?

Thanks for your help!

joshf

Some progress: adding decimation seems to directly reduce the artifacts, and with a decimationFactor of 4 they are almost unnoticeable. I don't really understand why, but at least the output now looks nearly perfect.

erik

Hi joshf ,
Decimation filter does remove quite a few points (bad ones as well), perhaps voxelization + removing statistical outliers would be beneficial instead. Apologies for the brightness fitler - it's on develop branch (PR here), so you could use it by checking out to develop branch and running python examples/isntall_requirements.py to get the wheel. Some folks also crop (or invalidate == set to 0) a few pixels around the edge of the depth frame, as you usually have bad pixels there. I hope this helps!
Thanks, Erik

joshf

Hi erik ,

We really appreciate your responses so far. We've been having similar issues on a different scene:

It should be a relatively flat plane, but the depth image is very much not:

But the dot grid and rectified images look fine:

(Here's the point cloud after post-processing, for clarity)

We haven't tried the brightness filter yet, but nothing looks over- or under-exposed (and we're trying to get correct readings on the toothbrush heads, not filter them out entirely); do you have any sense of what could be going wrong?

erik

Hi joshf ,
Could you share a recording (maybe with depthai-sdk) of this, so we can check it out locally? It's hard to suggest much useful from a few screenshots in such cases.
Thanks, Erik

joshf

erik
Will do, do you have a preferred file type & would there be anything is particular that it would be helpful to include (besides just a stream of the problematic scene)?
Thanks! -Josh

joshf

Hi erik ,

This is a zip file of the problematic scene, taken with the code below:

from depthai_sdk import OakCamera, RecordType

with OakCamera() as oak:
    color = oak.create_camera('color', resolution='1080P', fps=20, encode='H265')
    left = oak.create_camera('left', resolution='400p', fps=20, encode='H265')
    right = oak.create_camera('right', resolution='400p', fps=20, encode='H265')

    # Synchronize & save all (encoded) streams
    oak.record([color.out.encoded, left.out.encoded, right.out.encoded], './', RecordType.VIDEO)
    # Show color stream
    oak.visualize([color.out.camera], scale=2/3, fps=True)

    oak.start(blocking=True)

We appreciate your help!

Thanks,
Josh

joshf

(mp4 recording was hanging forever with this output:

x265 [info]: HEVC encoder version 3.5+1-f0c1022b6
x265 [info]: build info [Linux][GCC 10.2.1][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: Main profile, Level-3 (Main tier)
x265 [info]: Thread pool created using 16 threads
x265 [info]: Slices                              : 1
x265 [info]: frame threads / pool features       : 4 / wpp(8 rows)
x265 [warning]: Source height < 720p; disabling lookahead-slices
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge         : hex / 57 / 2 / 3
x265 [info]: Keyframe min / max / scenecut / bias  : 20 / 250 / 40 / 5.00 
x265 [info]: Lookahead / bframes / badapt        : 20 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb       : 1 / 1 / 0
x265 [info]: References / ref-limit  cu / depth  : 3 / off / on
x265 [info]: AQ: mode / str / qg-size / cu-tree  : 2 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress            : ABR-1024 kbps / 0.60
x265 [info]: tools: rd=3 psy-rd=2.00 early-skip rskip mode=1 signhide tmvp
x265 [info]: tools: b-intra strong-intra-smoothing deblock sao
x265 [info]: HEVC encoder version 3.5+1-f0c1022b6
x265 [info]: build info [Linux][GCC 10.2.1][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: Main profile, Level-3 (Main tier)
x265 [info]: Thread pool created using 16 threads
x265 [info]: Slices                              : 1
x265 [info]: frame threads / pool features       : 4 / wpp(8 rows)
x265 [warning]: Source height < 720p; disabling lookahead-slices
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge         : hex / 57 / 2 / 3
x265 [info]: Keyframe min / max / scenecut / bias  : 20 / 250 / 40 / 5.00 
x265 [info]: Lookahead / bframes / badapt        : 20 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb       : 1 / 1 / 0
x265 [info]: References / ref-limit  cu / depth  : 3 / off / on
x265 [info]: AQ: mode / str / qg-size / cu-tree  : 2 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress            : ABR-1024 kbps / 0.60
x265 [info]: tools: rd=3 psy-rd=2.00 early-skip rskip mode=1 signhide tmvp
x265 [info]: tools: b-intra strong-intra-smoothing deblock sao
x265 [info]: HEVC encoder version 3.5+1-f0c1022b6
x265 [info]: build info [Linux][GCC 10.2.1][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: Main profile, Level-3 (Main tier)
x265 [info]: Thread pool created using 16 threads
x265 [info]: Slices                              : 1
x265 [info]: frame threads / pool features       : 4 / wpp(8 rows)
x265 [warning]: Source height < 720p; disabling lookahead-slices
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge         : hex / 57 / 2 / 3
x265 [info]: Keyframe min / max / scenecut / bias  : 20 / 250 / 40 / 5.00 
x265 [info]: Lookahead / bframes / badapt        : 20 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb       : 1 / 1 / 0
x265 [info]: References / ref-limit  cu / depth  : 3 / off / on
x265 [info]: AQ: mode / str / qg-size / cu-tree  : 2 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress            : ABR-1024 kbps / 0.60
x265 [info]: tools: rd=3 psy-rd=2.00 early-skip rskip mode=1 signhide tmvp
x265 [info]: tools: b-intra strong-intra-smoothing deblock sao

)

joshf

We've also confirmed that this is a problem with that particular camera, not with the scene: it looks fine with a different OakD of the same model.

erik

Hi joshf ,
Perhaps it was a bad calibration with the current OAK, or the device got somehow damaged (a bit of bending) and calibration got invalid? Have you also tried recalibrating the device? Regarding the recording - some frames were dropped, frames are unusable (as using h26x).
Thanks, Erik