Depth image quality improvement suggestions and ideas

Robotik123

Hi,
I wanted to have a discussion on how to improve the depth image quality filtering out noise using depth ai v2 ros packages. I have looked into the documentation, where brightness filter, median filter, speckle filter, spatial filter, and ir dot projector are used. I tried to use them, at least configured them, to the best of my understanding. But reflections and bright daylight, are a major problem.. I have attached an rgb image and corresponding depth image. Any initial suggestions or ideas I am willing to know and understand.

Thanks 🙂

Oskar-Luxonis

Hey Robotik123 - reflections and the low-texture/glossy floor are difficult for stereo matching, so filters can reject or smooth incorrect depth but cannot make those measurements reliable. I’d first inspect the rectified left/right mono streams for saturation or exposure issues, then enable LR-check and use a stricter confidence threshold to prioritize valid depth over fill rate.

For filtering, use a depth-range threshold for your working distance, followed by speckle filtering. Be careful with spatial/hole-filling filters, as they can fill invalid regions with neighbouring depth values. The brightness filter is useful for invalidating saturated pixels/highlights, but it cannot recover their depth.

If you have a Pro device, the dot projector can help the textureless floor indoors; it will be much less effective in direct sunlight. Could you share your exact OAK model, DepthAI/ROS version, and stereo YAML/configuration?

Useful references:

Thanks,
Oskar

Robotik123

Yes I do have the oakd pro device. I am using Depthai v2 ros driver v2.12.2-jazzy-2-g3170e87, along with the depthai core 2.30.0-1noble.20250424.110307

I am using the active stereo vision for texture less floor and walls. Before we begin this discussion, I would like to note a few interesting observations:

when we implement the brightness filter a bit too aggressively to filter out the highly bright pixels of the depth image, like setting the max value of brightness as 220. It filters out the brightest pixels in every depth image frame, it could also be dot projector bright patterns(1000) on the walls or the floor.
I am currently using the default setting as high accuracy, which I see an increased range as compared to high detail but then high detail has a better capability of handling the reflections sacrificing the range.

For the mono cameras, we have default parameters set, also the fps of the cameras is significantly low around 5fps, due to application of the filters, which we might want want to consider if we are setting the exposure and iso parameter values.the parameters are as follows:

/oakd:
  ros__parameters:
    camera:
      i_calibration_dump: false
      i_enable_diagnostics: true
      i_enable_imu: true
      i_enable_ir: true
      i_enable_sync: true
      i_external_calibration_path: ''
      i_floodlight_brightness: 0
      i_ip: ''
      i_laser_dot_brightness: 1000
      i_mx_id: ''
      i_nn_type: none
      i_pipeline_dump: false
      i_pipeline_type: RGBD
      i_publish_tf_from_calibration: false
      i_restart_on_diagnostics_error: false
      i_rs_compat: false
      i_tf_base_frame: oak
      i_tf_cam_pitch: '0.0'
      i_tf_cam_pos_x: '0.0'
      i_tf_cam_pos_y: '0.0'
      i_tf_cam_pos_z: '0.0'
      i_tf_cam_roll: '0.0'
      i_tf_cam_yaw: '0.0'
      i_tf_camera_model: ''
      i_tf_camera_name: oakd
      i_tf_custom_urdf_location: ''
      i_tf_custom_xacro_args: ''
      i_tf_imu_from_descr: 'false'
      i_tf_parent_frame: oak-d-base-frame
      i_usb_port_id: ''
      i_usb_speed: SUPER_PLUS
    diagnostic_updater:
      period: 1.0
      use_fqn: false
    imu:
      i_acc_cov: 0.0
      i_acc_freq: 400
      i_acc_mode: ACCELEROMETER_RAW
      i_batch_report_threshold: 5
      i_enable_acc: true
      i_enable_gyro: true
      i_enable_mag: true
      i_enable_rotation: true
      i_get_base_device_timestamp: false
      i_gyro_cov: 0.0
      i_gyro_freq: 400
      i_gyro_mode: GYROSCOPE_RAW
      i_mag_cov: 0.0
      i_mag_freq: 100
      i_mag_mode: MAGNETOMETER_RAW
      i_max_batch_reports: 10
      i_max_q_size: 8
      i_message_type: IMU
      i_rot_cov: 0.0
      i_rot_freq: 400
      i_rot_mode: ROTATION_VECTOR
      i_sync_method: LINEAR_INTERPOLATE_ACCEL
      i_update_ros_base_time_on_ros_msg: false
    left:
      i_add_exposure_offset: false
      i_board_socket_id: 1
      i_calibration_file: ''
      i_disable_node: false
      i_enable_feature_tracker: false
      i_enable_lazy_publisher: true
      i_enable_nn: false
      i_exposure_offset: 0
      i_flip_published_image: false
      i_fps: 8.0
      i_fsync_continuous: false
      i_fsync_trigger: false
      i_get_base_device_timestamp: false
      i_height: 720
      i_low_bandwidth: false
      i_low_bandwidth_bitrate: 0
      i_low_bandwidth_ffmpeg_encoder: libx264
      i_low_bandwidth_frame_freq: 30
      i_low_bandwidth_profile: 4
      i_low_bandwidth_quality: 50
      i_max_q_size: 30
      i_publish_compressed: false
      i_publish_topic: false
      i_resolution: 720P
      i_reverse_stereo_socket_order: false
      i_sensor_img_orientation: AUTO
      i_set_isp3a_fps: false
      i_simulate_from_topic: false
      i_simulated_topic_name: ''
      i_synced: true
      i_update_ros_base_time_on_ros_msg: false
      i_width: 1280
      r_auto_exp_region_height: 0
      r_auto_exp_region_start_x: 0
      r_auto_exp_region_start_y: 0
      r_auto_exp_region_width: 0
      r_auto_exposure_limit: 1000
      r_chroma_denoise: 1
      r_exposure: 1000
      r_iso: 800
      r_luma_denoise: 1
      r_set_auto_exp_region: false
      r_set_auto_exposure_limit: false
      r_set_chroma_denoise: false
      r_set_luma_denoise: false
      r_set_man_exposure: false
      r_set_sharpness: false
      r_sharpness: 1
    oakd:
      rgb:
        image_raw:
          compressed:
            format: jpeg
            jpeg_quality: 95
            png_level: 3
            tiff:
              res_unit: inch
              xdpi: -1
              ydpi: -1
          compressedDepth:
            depth_max: 10.0
            depth_quantization: 100.0
            format: png
            png_level: 3
          depth_max: 10.0
          depth_quantization: 100.0
          enable_pub_plugins:
          - image_transport/compressedDepth
          - image_transport/compressed
          - image_transport/raw
          - image_transport/theora
          - image_transport/zstd
          format: jpeg
          jpeg_quality: 95
          keyframe_frequency: 64
          optimize_for: true
          png_level: 3
          quality: 31
          target_bitrate: 800000
          theora:
            keyframe_frequency: 64
            optimize_for: true
            quality: 31
            target_bitrate: 800000
          tiff:
            res_unit: inch
            xdpi: -1
            ydpi: -1
          zstd:
            zstd_level: 3
      right:
        image_raw:
          compressed:
            format: jpeg
            jpeg_quality: 95
            png_level: 3
            tiff:
              res_unit: inch
              xdpi: -1
              ydpi: -1
          compressedDepth:
            depth_max: 10.0
            depth_quantization: 100.0
            format: png
            png_level: 3
          depth_max: 10.0
          depth_quantization: 100.0
          enable_pub_plugins:
          - image_transport/compressedDepth
          - image_transport/compressed
          - image_transport/raw
          - image_transport/theora
          - image_transport/zstd
          format: jpeg
          jpeg_quality: 95
          keyframe_frequency: 64
          optimize_for: true
          png_level: 3
          quality: 31
          target_bitrate: 800000
          theora:
            keyframe_frequency: 64
            optimize_for: true
            quality: 31
            target_bitrate: 800000
          tiff:
            res_unit: inch
            xdpi: -1
            ydpi: -1
          zstd:
            zstd_level: 3
      stereo:
        image_raw:
          compressed:
            format: jpeg
            jpeg_quality: 95
            png_level: 3
            tiff:
              res_unit: inch
              xdpi: -1
              ydpi: -1
          compressedDepth:
            depth_max: 10.0
            depth_quantization: 100.0
            format: png
            png_level: 3
          depth_max: 10.0
          depth_quantization: 100.0
          enable_pub_plugins:
          - image_transport/compressedDepth
          - image_transport/compressed
          - image_transport/raw
          - image_transport/theora
          - image_transport/zstd
          format: jpeg
          jpeg_quality: 95
          keyframe_frequency: 64
          optimize_for: true
          png_level: 3
          quality: 31
          target_bitrate: 800000
          theora:
            keyframe_frequency: 64
            optimize_for: true
            quality: 31
            target_bitrate: 800000
          tiff:
            res_unit: inch
            xdpi: -1
            ydpi: -1
          zstd:
            zstd_level: 3
    pipeline_gen:
      i_enable_diagnostics: true
      i_enable_imu: true
      i_enable_sync: true
    qos_overrides:
      /parameter_events:
        publisher:
          depth: 1000
          durability: volatile
          history: keep_last
          reliability: reliable
    rgb:
      i_add_exposure_offset: false
      i_board_socket_id: 0
      i_calibration_file: ''
      i_color_order: BGR
      i_disable_node: false
      i_enable_feature_tracker: false
      i_enable_lazy_publisher: true
      i_enable_nn: false
      i_enable_preview: false
      i_exposure_offset: 0
      i_flip_published_image: false
      i_fps: 8.0
      i_fsync_continuous: false
      i_fsync_trigger: false
      i_get_base_device_timestamp: false
      i_height: 360
      i_interleaved: false
      i_isp_den: 3
      i_isp_num: 1
      i_keep_preview_aspect_ratio: false
      i_low_bandwidth: false
      i_low_bandwidth_bitrate: 0
      i_low_bandwidth_ffmpeg_encoder: libx264
      i_low_bandwidth_frame_freq: 30
      i_low_bandwidth_profile: 4
      i_low_bandwidth_quality: 50
      i_max_q_size: 30
      i_output_isp: true
      i_preview_height: 480
      i_preview_size: 300
      i_preview_width: 640
      i_publish_compressed: false
      i_publish_topic: true
      i_resolution: 1080P
      i_reverse_stereo_socket_order: false
      i_sensor_img_orientation: AUTO
      i_set_isp3a_fps: false
      i_set_isp_scale: true
      i_simulate_from_topic: false
      i_simulated_topic_name: ''
      i_synced: true
      i_update_ros_base_time_on_ros_msg: false
      i_width: 640
      r_auto_exp_region_height: 0
      r_auto_exp_region_start_x: 0
      r_auto_exp_region_start_y: 0
      r_auto_exp_region_width: 0
      r_auto_exposure_limit: 1000
      r_chroma_denoise: 1
      r_exposure: 20000
      r_focus: 130
      r_iso: 800
      r_luma_denoise: 1
      r_set_auto_exp_region: false
      r_set_auto_exposure_limit: false
      r_set_chroma_denoise: false
      r_set_luma_denoise: false
      r_set_man_exposure: false
      r_set_man_focus: true
      r_set_man_whitebalance: false
      r_set_sharpness: false
      r_sharpness: 1
      r_whitebalance: 3300
    right:
      i_add_exposure_offset: false
      i_board_socket_id: 2
      i_calibration_file: ''
      i_disable_node: false
      i_enable_feature_tracker: false
      i_enable_lazy_publisher: true
      i_enable_nn: false
      i_exposure_offset: 0
      i_flip_published_image: false
      i_fps: 8.0
      i_fsync_continuous: false
      i_fsync_trigger: false
      i_get_base_device_timestamp: false
      i_height: 720
      i_low_bandwidth: false
      i_low_bandwidth_bitrate: 0
      i_low_bandwidth_ffmpeg_encoder: libx264
      i_low_bandwidth_frame_freq: 30
      i_low_bandwidth_profile: 4
      i_low_bandwidth_quality: 50
      i_max_q_size: 30
      i_publish_compressed: false
      i_publish_topic: true
      i_resolution: 720P
      i_reverse_stereo_socket_order: false
      i_sensor_img_orientation: AUTO
      i_set_isp3a_fps: false
      i_simulate_from_topic: false
      i_simulated_topic_name: ''
      i_synced: true
      i_update_ros_base_time_on_ros_msg: false
      i_width: 1280
      r_auto_exp_region_height: 0
      r_auto_exp_region_start_x: 0
      r_auto_exp_region_start_y: 0
      r_auto_exp_region_width: 0
      r_auto_exposure_limit: 1000
      r_chroma_denoise: 1
      r_exposure: 1000
      r_iso: 800
      r_luma_denoise: 1
      r_set_auto_exp_region: false
      r_set_auto_exposure_limit: false
      r_set_chroma_denoise: false
      r_set_luma_denoise: false
      r_set_man_exposure: false
      r_set_sharpness: false
      r_sharpness: 1
    start_type_description_service: true
    stereo:
      i_add_exposure_offset: false
      i_align_depth: true
      i_bilateral_sigma: 0
      i_board_socket_id: 0
      i_brightness_filter_min_brightness: 0
      i_brightness_filter_max_brightness: 220
      i_calibration_file: ''
      i_depth_filter_size: 3
      i_depth_preset: HIGH_ACCURACY
      i_disparity_width: DISPARITY_96
      i_enable_alpha_scaling: false
      i_enable_brightness_filter: true
      i_enable_companding: false
      i_enable_decimation_filter: false
      i_enable_disparity_shift: false
      i_enable_distortion_correction: false
      i_enable_lazy_publisher: true
      i_enable_spatial_filter: true
      i_enable_spatial_nn: false
      i_enable_speckle_filter: true
      i_enable_temporal_filter: false
      i_enable_threshold_filter: false
      i_exposure_offset: 0
      i_extended_disp: false
      i_flip_published_image: false
      i_get_base_device_timestamp: false
      i_height: 360
      i_left_rect_add_exposure_offset: false
      i_left_rect_enable_feature_tracker: false
      i_left_rect_exposure_offset: 0
      i_left_rect_low_bandwidth: false
      i_left_rect_low_bandwidth_bitrate: 0
      i_left_rect_low_bandwidth_ffmpeg_encoder: libx264
      i_left_rect_low_bandwidth_frame_freq: 30
      i_left_rect_low_bandwidth_profile: 4
      i_left_rect_low_bandwidth_quality: 50
      i_left_rect_publish_compressed: false
      i_left_rect_publish_topic: false
      i_left_rect_synced: false
      i_left_socket_id: 1
      i_low_bandwidth: false
      i_low_bandwidth_bitrate: 0
      i_low_bandwidth_ffmpeg_encoder: libx264
      i_low_bandwidth_frame_freq: 30
      i_low_bandwidth_profile: 4
      i_low_bandwidth_quality: 50
      i_lr_check: true
      i_lrc_threshold: 7
      i_max_q_size: 30
      i_output_disparity: false
      i_publish_compressed: false
      i_publish_synced_rect_pair: false
      i_publish_topic: true
      i_rectify_edge_fill_color: 0
      i_reverse_stereo_socket_order: false
      i_right_rect_add_exposure_offset: false
      i_right_rect_enable_feature_tracker: false
      i_right_rect_exposure_offset: 0
      i_right_rect_low_bandwidth: false
      i_right_rect_low_bandwidth_bitrate: 0
      i_right_rect_low_bandwidth_ffmpeg_encoder: libx264
      i_right_rect_low_bandwidth_frame_freq: 30
      i_right_rect_low_bandwidth_profile: 4
      i_right_rect_low_bandwidth_quality: 50
      i_right_rect_publish_compressed: false
      i_right_rect_publish_topic: false
      i_right_rect_synced: false
      i_right_socket_id: 2
      i_set_disparity_to_depth_use_spec_translation: false
      i_set_input_size: false
      i_socket_name: rgb
      i_spatial_filter_alpha: 0.3
      i_spatial_filter_delta: 15
      i_spatial_filter_hole_filling_radius: 1
      i_spatial_filter_iterations: 1
      i_spatial_nn_source: right
      i_speckle_filter_speckle_range: 50
      i_stereo_conf_threshold: 240
      i_subpixel: true
      i_subpixel_fractional_bits: 3
      i_synced: true
      i_update_ros_base_time_on_ros_msg: false
      i_width: 640
    sync:
      sync_attempts: 10
      sync_threshold: 10
    use_sim_time: false

Oskar-Luxonis

Hey Robotik123 - thanks, this clarifies it. With i_brightness_filter_max_brightness: 220, the filter can invalidate the bright IR-dot pattern as well as saturated highlights. For an indoor active-stereo baseline, please try disabling the brightness filter (or raising its max to 255/256) first.

Also, your r_exposure and r_iso values are not currently active, as r_set_man_exposure is false for both mono cameras. You are therefore using auto exposure. If you want to tune exposure/ISO, enable that flag for both cameras, or enable the auto-exposure limit instead.

Finally, i_stereo_conf_threshold: 240 overrides the High Accuracy preset and favors depth density. Try 200 first; lowering it further will reject more unreliable reflective-surface depth at the cost of fill rate.

Your High Detail observation is likely influenced by its temporal filtering and decimation, so it is smoothing the result rather than making reflections intrinsically measurable.

Thanks,
Oskar

Robotik123

Hey @Oskar-Luxonis,
a quick addition to your response, when i implement the high accuracy preset, in my yaml file i generally set temporal filter to false, and also turn off decimation, and conf threshold is 240 same as high accuracy. Also the median filter was forcefully turned off by driver itself during runtime stating the following:

[component_container-11] [INFO] [1786968941.384005893] [oakd]: Camera ready!
[component_container-11] [18443010F1FD990F00] [1.1.2.2] [2.381] [StereoDepth(6)] [error] Maximum disparity value '1520' exceeds the maximum supported '1024' by median filter. Disabling median filter!
[component_container-11] [18443010F1FD990F00] [1.1.2.2] [2.212] [StereoDepth(6)] [error] Maximum disparity value '1520' exceeds the maximum supported '1024' by median filter. Disabling median filter!

so with subpixel bits:3 following is the depth image corresponding to it, as an addition I would like to mention, the parameters are exactly same as my previous comment with just the high detail enabled and brightness filter off.

Also, if you could mention some starting values for exposure and iso would be great, my understanding about these specific params is not that great to be honest.

I will be testing out the remaining brightness filter param options.

Thanks for your support! 🙂

Robotik123

Hey, @Oskar-Luxonis

With High accuracy default preset: I tried switching off the brightness filter, that did two things improved the fps as we have one reduced filter in the pipeline, but the reflections along the floor were present which were earlier blacked out by the by the brightness filter.

If you could kindly guide me with the iso and exposure settings would be great help, there is also an option auto exposure limit, which we can set. Regarding the anti banding settings, what is the default value in generic camera launch file, I dont see that parameter exposed in the parameter yaml file list here: https://docs.luxonis.com/software/ros/depthai-ros/driver/

Thanks! 🙂

Oskar-Luxonis

Hey Robotik123 - that behavior is expected: without the brightness filter, reflective-floor pixels are no longer invalidated, so you see the underlying incorrect stereo matches. For reflections, invalid/black depth is generally safer than a plausible but incorrect value.

The median-filter warning is also expected with Extended Disparity + 3-bit subpixel: the maximum disparity becomes 1520, above the median hardware limit of 1024, so it is disabled. Please leave median off in this mode.

One correction: i_stereo_conf_threshold: 240 is not the High Accuracy value—it overrides it. High Accuracy uses 200, so I would test 200 first to reject more unreliable matches.

For exposure, I suggest keeping auto exposure initially and enabling r_set_auto_exposure_limit: true for both mono cameras, with r_auto_exposure_limit: 10000 (10 ms) as a starting point. This will avoid excessively long exposures; then inspect the rectified mono images and adjust from there. Your configured manual exposure/ISO will only take effect once r_set_man_exposure: true.

Anti-banding is 50 Hz by default and is not currently exposed by the ROS driver YAML.

Thanks,
Oskar

Robotik123

Hey Oskar-Luxonis ,
I have an additional question, i see the ideal range for oakd pro is 80cm to 12 metres, but still i see depth values like 50-60 metres. Is it not something not discarded automatically past the ideal range?

Thanks!

Oskar-Luxonis

Hey Robotik123 - the 80 cm to 12 m range is the range where we specify depth performance. It is not a hard output limit. Stereo can still produce non-zero values at 50–60 m, but these values can have large errors because disparity is very small at that distance.

Your threshold filter is disabled. Enable it to discard values outside your required range, for example:

i_enable_threshold_filter: true
i_threshold_filter_min_range: 800
i_threshold_filter_max_range: 12000

This keeps depth values from 0.8 m to 12 m. Values outside this range become invalid (0).

Thanks,
Oskar

Robotik123

hey, Oskar-Luxonis

Edit: my mistake 🙂 the viewer i am using had turned on some sort of rectification automatically, disabling it solved the problem.

Quick question, is it possible to use the neural assisted stereo depth, that oak4 cameras use on board. I understand currently they are being rolled for rvc4 cameras, oakd pro camera can also do that with not running them on board but running them off board on a powerful compute. If luxonis team could assist me on that, would be really helpful!

https://discuss.luxonis.com/blog/6656-neural-assisted-stereo-classical-stereo-enhanced-by-neural-depth

Thanks!

Oskar-Luxonis

Hey Robotik123 - good that you found the viewer rectification setting.

Neural Assisted Stereo is available only on RVC4 devices with DepthAI v3. It cannot run on OAK-D Pro (RVC2), and it has no supported off-device version.

For OAK-D Pro with a powerful host computer, you can run a separate neural stereo model on the rectified left and right images. Our Foundation Stereo example is a good starting point. It runs inference on the host and compares its result with device stereo depth.

Thanks,
Oskar

Robotik123

Hey Oskar-Luxonis ,

I am having a difficulty in understanding the RGBstereo settings,
I want the rectified mono images for the foundation stereo, but I dont see them, under the stereo topics published. where are they getting published? I dont want the raw mono images, i want the rectified ones?

/oakd:
  ros__parameters:

    camera:
      i_enable_ir: true
      i_floodlight_brightness: 0
      i_laser_dot_brightness: 1000
      i_nn_type: none
      i_pipeline_type: RGBStereo
      i_usb_speed: SUPER_PLUS
      i_enable_sync: true

    pipeline_gen:
      i_enable_sync: true

    rgb:
      i_board_socket_id: 0
      i_fps: 15.0
      i_interleaved: false
      
      i_output_isp: true
      
      i_isp_den: 3
      i_isp_num: 1

      i_enable_preview: false
      i_preview_width: 640
      i_preview_height: 360
      i_keep_preview_aspect_ratio: false

      i_low_bandwidth: false
      i_publish_topic: true

      i_resolution: '1080P'
      
      r_set_man_focus: true
      r_focus: 120
      r_set_man_exposure: false
      r_set_auto_exposure_limit: true
      r_auto_exposure_limit: 15000

      i_synced: true
      i_get_base_device_timestamp: false

    right:
      i_publish_topic: true
      i_synced: true
      i_fps: 15.0
      r_set_man_exposure: false
      r_set_auto_exposure_limit: false
      r_auto_exposure_limit: 10000
      
    left:
      i_publish_topic: true
      i_synced: true
      i_fps: 15.0
      r_set_man_exposure: false
      r_set_auto_exposure_limit: false
      r_auto_exposure_limit: 10000

    stereo:
      i_fps: 15.0
      i_width: 640
      i_height: 360
      
      i_enable_distortion_correction: true
      i_publish_left_rect: true
      i_publish_right_rect: true
      i_publish_synced_rect_pair: true
      
      i_left_rect_synced: true
      i_right_rect_synced: true
      i_set_input_size: false
      
      i_get_base_device_timestamp: false

    imu:
      i_get_base_device_timestamp: false
    
    sync:
      sync_attempts: 10
      sync_threshold: 10


    use_sim_time: false

also if u could share some info on the i_enable_distortion_correction: true

what kind of distortion correction does it do, i thought previously during my depth estimation when using rgbd pipeline, this is done automatically. I see this false as default in dev options yaml file in depth ai v2 ros.

Thanks!

Oskar-Luxonis

Hey Robotik123 - Please use these parameter names:

i_left_rect_publish_topic: true
i_right_rect_publish_topic: true

The rectified images publish on /oakd/left/image_rect and /oakd/right/image_rect, not under the stereo topic. i_publish_synced_rect_pair: true synchronizes these two topics; it does not create one combined topic.

i_enable_distortion_correction: true uses the calibration data to correct mono-lens distortion before stereo matching. Stereo rectification is already part of depth processing; this setting selects the more complete mesh-based correction. It does not undistort the RGB stream.

Thanks,
Oskar