Hi all,
We're seeing major artifacts in the depth images from our OAK-D Pro POE, along edges in the scene -- has anyone seen something similar and/or know how to get rid of them? We've tried messing with all the settings we can think of and nothing has seemed relevant.

OAK-D point cloud:

Actual scene:

  • erik replied to this.

    Hi joshf ,
    It's a bit hard to understand the perception, which artifacts are you referring to? At first, it looks like subpixel mode would remove the "layering", and perhaps a brightness filter for the edged of depth frame (which has invalid depth due to rectification). Also could you share rectified images? What's the power output on dot projector?
    Thanks, Erik

      Hi erik ,

      I appreciate the response. The artifacts I'm referring to are at the edges of the cardboard box and between the rows of cereal boxes, especially in this area:

      Would subpixel mode or brightness filtering help, in your opinion?

      We've tried changing brightness from max to min and various intermediate values, without any noticeable change. The full camera settings we used for these images are as follows:

              monoResolution = dai.MonoCameraProperties.SensorResolution.THE_400_P
      
              left = self.pipeline.create(dai.node.MonoCamera)
              left.setResolution(monoResolution)
              left.setBoardSocket(dai.CameraBoardSocket.LEFT)
              left.setFps(fps)
      
              right = self.pipeline.create(dai.node.MonoCamera)
              right.setResolution(monoResolution)
              right.setBoardSocket(dai.CameraBoardSocket.RIGHT)
              right.setFps(fps)
      
              stereo = self.pipeline.create(dai.node.StereoDepth)
              stereo.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.HIGH_DENSITY)
              stereo.setPostProcessingHardwareResources(
                  9, 6
              ) 
      
              stereo.setExtendedDisparity(True) 
              stereo.initialConfig.setMedianFilter(
                  dai.StereoDepthProperties.MedianFilter.KERNEL_7x7  # not technically compatible with extended disparity but hasn't made a big difference
              )
              stereo.initialConfig.setConfidenceThreshold(200) 
              stereo.initialConfig.setDepthUnit(
                  dai.RawStereoDepthConfig.AlgorithmControl.DepthUnit.MILLIMETER
              )
      
              left.out.link(stereo.left)
              right.out.link(stereo.right)
      
              config = stereo.initialConfig.get()
      
              config.postProcessing.speckleFilter.enable = True
              config.postProcessing.speckleFilter.speckleRange = 100
      
              config.postProcessing.temporalFilter.enable = True
      
              config.postProcessing.spatialFilter.enable = True
              config.postProcessing.spatialFilter.holeFillingRadius = 5
              config.postProcessing.spatialFilter.numIterations = 1
      
              config.postProcessing.thresholdFilter.minRange = int(
                 .15 * 1000
              )  
              config.postProcessing.thresholdFilter.maxRange = int(
                  1.5 * 1000
              )
      
              ...
              device = dai.Device(ip.value)
              # device.setIrLaserDotProjectorBrightness(765)

      Rectified images from depthai_demo:

      These are with subpixel on and max dot brightness: the artifacts have less layering, but they are still very much present:

      • erik replied to this.

        Hi joshf ,
        I would first try the brightness filter, and enabling dot projector - it looks like it's disabled, and that texture isn't that great. In some parts it's overexposed, in some parts there isn't enough texture.
        Could you also share the pointcloud itself?
        Thanks, Erik

          Hi erik ,
          Where do i find the brightness filter? I tried searching the docs and didn't come across anything.
          Is there a particular format which would be convenient for me to upload the point cloud?

          Thanks for your help!

          Some progress: adding decimation seems to directly reduce the artifacts, and with a decimationFactor of 4 they are almost unnoticeable. I don't really understand why, but at least the output now looks nearly perfect.

          • erik replied to this.

            Hi joshf ,
            Decimation filter does remove quite a few points (bad ones as well), perhaps voxelization + removing statistical outliers would be beneficial instead. Apologies for the brightness fitler - it's on develop branch (PR here), so you could use it by checking out to develop branch and running python examples/isntall_requirements.py to get the wheel. Some folks also crop (or invalidate == set to 0) a few pixels around the edge of the depth frame, as you usually have bad pixels there. I hope this helps!
            Thanks, Erik

              Hi erik ,

              We really appreciate your responses so far. We've been having similar issues on a different scene:

              It should be a relatively flat plane, but the depth image is very much not:

              But the dot grid and rectified images look fine:

              (Here's the point cloud after post-processing, for clarity)

              We haven't tried the brightness filter yet, but nothing looks over- or under-exposed (and we're trying to get correct readings on the toothbrush heads, not filter them out entirely); do you have any sense of what could be going wrong?

              • erik replied to this.

                Hi joshf ,
                Could you share a recording (maybe with depthai-sdk) of this, so we can check it out locally? It's hard to suggest much useful from a few screenshots in such cases.
                Thanks, Erik

                  erik
                  Will do, do you have a preferred file type & would there be anything is particular that it would be helpful to include (besides just a stream of the problematic scene)?
                  Thanks! -Josh

                  Hi erik ,

                  This is a zip file of the problematic scene, taken with the code below:

                  from depthai_sdk import OakCamera, RecordType
                  
                  with OakCamera() as oak:
                      color = oak.create_camera('color', resolution='1080P', fps=20, encode='H265')
                      left = oak.create_camera('left', resolution='400p', fps=20, encode='H265')
                      right = oak.create_camera('right', resolution='400p', fps=20, encode='H265')
                  
                      # Synchronize & save all (encoded) streams
                      oak.record([color.out.encoded, left.out.encoded, right.out.encoded], './', RecordType.VIDEO)
                      # Show color stream
                      oak.visualize([color.out.camera], scale=2/3, fps=True)
                  
                      oak.start(blocking=True)

                  We appreciate your help!

                  Thanks,
                  Josh

                  (mp4 recording was hanging forever with this output:

                  x265 [info]: HEVC encoder version 3.5+1-f0c1022b6
                  x265 [info]: build info [Linux][GCC 10.2.1][64 bit] 8bit+10bit+12bit
                  x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
                  x265 [info]: Main profile, Level-3 (Main tier)
                  x265 [info]: Thread pool created using 16 threads
                  x265 [info]: Slices                              : 1
                  x265 [info]: frame threads / pool features       : 4 / wpp(8 rows)
                  x265 [warning]: Source height < 720p; disabling lookahead-slices
                  x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
                  x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
                  x265 [info]: ME / range / subpel / merge         : hex / 57 / 2 / 3
                  x265 [info]: Keyframe min / max / scenecut / bias  : 20 / 250 / 40 / 5.00 
                  x265 [info]: Lookahead / bframes / badapt        : 20 / 4 / 2
                  x265 [info]: b-pyramid / weightp / weightb       : 1 / 1 / 0
                  x265 [info]: References / ref-limit  cu / depth  : 3 / off / on
                  x265 [info]: AQ: mode / str / qg-size / cu-tree  : 2 / 1.0 / 32 / 1
                  x265 [info]: Rate Control / qCompress            : ABR-1024 kbps / 0.60
                  x265 [info]: tools: rd=3 psy-rd=2.00 early-skip rskip mode=1 signhide tmvp
                  x265 [info]: tools: b-intra strong-intra-smoothing deblock sao
                  x265 [info]: HEVC encoder version 3.5+1-f0c1022b6
                  x265 [info]: build info [Linux][GCC 10.2.1][64 bit] 8bit+10bit+12bit
                  x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
                  x265 [info]: Main profile, Level-3 (Main tier)
                  x265 [info]: Thread pool created using 16 threads
                  x265 [info]: Slices                              : 1
                  x265 [info]: frame threads / pool features       : 4 / wpp(8 rows)
                  x265 [warning]: Source height < 720p; disabling lookahead-slices
                  x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
                  x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
                  x265 [info]: ME / range / subpel / merge         : hex / 57 / 2 / 3
                  x265 [info]: Keyframe min / max / scenecut / bias  : 20 / 250 / 40 / 5.00 
                  x265 [info]: Lookahead / bframes / badapt        : 20 / 4 / 2
                  x265 [info]: b-pyramid / weightp / weightb       : 1 / 1 / 0
                  x265 [info]: References / ref-limit  cu / depth  : 3 / off / on
                  x265 [info]: AQ: mode / str / qg-size / cu-tree  : 2 / 1.0 / 32 / 1
                  x265 [info]: Rate Control / qCompress            : ABR-1024 kbps / 0.60
                  x265 [info]: tools: rd=3 psy-rd=2.00 early-skip rskip mode=1 signhide tmvp
                  x265 [info]: tools: b-intra strong-intra-smoothing deblock sao
                  x265 [info]: HEVC encoder version 3.5+1-f0c1022b6
                  x265 [info]: build info [Linux][GCC 10.2.1][64 bit] 8bit+10bit+12bit
                  x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
                  x265 [info]: Main profile, Level-3 (Main tier)
                  x265 [info]: Thread pool created using 16 threads
                  x265 [info]: Slices                              : 1
                  x265 [info]: frame threads / pool features       : 4 / wpp(8 rows)
                  x265 [warning]: Source height < 720p; disabling lookahead-slices
                  x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
                  x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
                  x265 [info]: ME / range / subpel / merge         : hex / 57 / 2 / 3
                  x265 [info]: Keyframe min / max / scenecut / bias  : 20 / 250 / 40 / 5.00 
                  x265 [info]: Lookahead / bframes / badapt        : 20 / 4 / 2
                  x265 [info]: b-pyramid / weightp / weightb       : 1 / 1 / 0
                  x265 [info]: References / ref-limit  cu / depth  : 3 / off / on
                  x265 [info]: AQ: mode / str / qg-size / cu-tree  : 2 / 1.0 / 32 / 1
                  x265 [info]: Rate Control / qCompress            : ABR-1024 kbps / 0.60
                  x265 [info]: tools: rd=3 psy-rd=2.00 early-skip rskip mode=1 signhide tmvp
                  x265 [info]: tools: b-intra strong-intra-smoothing deblock sao

                  )

                  We've also confirmed that this is a problem with that particular camera, not with the scene: it looks fine with a different OakD of the same model.

                  • erik replied to this.

                    Hi joshf ,
                    Perhaps it was a bad calibration with the current OAK, or the device got somehow damaged (a bit of bending) and calibration got invalid? Have you also tried recalibrating the device? Regarding the recording - some frames were dropped, frames are unusable (as using h26x).
                    Thanks, Erik