Single frame missing depth

WouterOddBot

We had the camera running for a few hours in a static position. Rarely we encounter one single depth frame where half of the image is somehow missing, even when the mono left and right image are fine. Is there something we can do to fix this?

Normal depth frame:

Problematic depth frame:

Our depth config:

    depth = pipeline.createStereoDepth()
    depth.setDepthAlign(CameraBoardSocket.RIGHT)
    depth.setDefaultProfilePreset(StereoDepth.PresetMode.HIGH_ACCURACY)
    depth.setExtendedDisparity(False)  # no good for long range
    depth.setLeftRightCheck(True)  # required for depth alignment
    depth.enableDistortionCorrection(True)
    depth.initialConfig.setConfidenceThreshold(160)
    depth.initialConfig.setMedianFilter(StereoDepthProperties.MedianFilter.MEDIAN_OFF)
    depth.initialConfig.setSubpixel(True)
    depth.initialConfig.setSubpixelFractionalBits(5)
    depth.initialConfig.setBilateralFilterSigma(0)
    depth.initialConfig.setDisparityShift(0)
    depth.initialConfig.setDepthUnit(RawStereoDepthConfig.AlgorithmControl.DepthUnit.CUSTOM)
    depth_config = depth.initialConfig.get()
    depth_config.algorithmControl.customDepthUnitMultiplier = 5000
    depth_config.postProcessing.brightnessFilter.minBrightness = 16
    depth_config.postProcessing.brightnessFilter.maxBrightness = 240
    depth_config.postProcessing.decimationFilter.decimationFactor = 1  # disables decimation filter
    depth_config.postProcessing.decimationFilter.decimationMode = StereoDepthConfig.PostProcessing.DecimationFilter.DecimationMode.PIXEL_SKIPPING  # default
    depth_config.postProcessing.spatialFilter.enable = True
    depth_config.postProcessing.spatialFilter.alpha = 0.5  # default
    depth_config.postProcessing.spatialFilter.delta = 0  # default
    depth_config.postProcessing.spatialFilter.holeFillingRadius = 2  # default
    depth_config.postProcessing.spatialFilter.numIterations = 1  # default
    depth_config.postProcessing.speckleFilter.enable = True
    depth_config.postProcessing.speckleFilter.differenceThreshold = 2  # default
    depth_config.postProcessing.speckleFilter.speckleRange = 25
    depth_config.postProcessing.temporalFilter.enable = False
    depth_config.postProcessing.temporalFilter.alpha = 0.4  # default
    depth_config.postProcessing.temporalFilter.delta = 0  # default
    depth_config.postProcessing.temporalFilter.persistencyMode = StereoDepthConfig.PostProcessing.TemporalFilter.PersistencyMode.VALID_2_IN_LAST_4  # default
    depth_config.postProcessing.thresholdFilter.minRange = 0  # default
    depth_config.postProcessing.thresholdFilter.maxRange = 65536  # default
    depth_config.postProcessing.filteringOrder = [RawStereoDepthConfig.PostProcessing.Filter.MEDIAN,
                                                  RawStereoDepthConfig.PostProcessing.Filter.DECIMATION,
                                                  RawStereoDepthConfig.PostProcessing.Filter.SPECKLE,
                                                  RawStereoDepthConfig.PostProcessing.Filter.SPATIAL,
                                                  RawStereoDepthConfig.PostProcessing.Filter.TEMPORAL, ]  # default
    depth.initialConfig.set(depth_config)

OskarSonc

Hi WouterOddBot - this usually looks like a one-off bad stereo pair/sync hiccup rather than calibration, and if you’re viewing left/right from separate queues they might not be the exact pair used for that depth frame. Could you log getSequenceNum() (or timestamps) for depth and rectifiedLeft/Right coming out of the StereoDepth node on a “bad” frame, and let us know which device you’re using and whether it’s USB or Ethernet/PoE? Also worth trying is locking mono exposure/gain after AE settles and temporarily widening/disabling the brightnessFilter to see if it’s masking half the frame.

Thanks,
Oskar

WouterOddBot

Hi Oskar,

Thanks for your reply. We actually bypassed the problem using the temporal filter. Our camera moves very slow in a static environment, so for us it's safe to replace invalid pixels with those from a few frames old.

OskarSonc

WouterOddBot - glad you solve it!