Low and drop in FPS and unsynchronized

ShivamSharma

Hi,

I am experiencing low fps when I set i_fps = 90.0 or the default 30. I check the fps in Rviz which shows the fps of the rectified images to be 12.5hz. Also, the fps keeps on decreasing slowly and the left and right images are unsynchronized after some time.

In rviz I can see sometimes that the image messages pause for 1.5 seconds when I have launched the ROS 2 driver. Please help me get a good fps and synchronization so that slam can perform better.

The following is my config file:

/oak:
  ros__parameters:
    camera:
      i_enable_imu: true
      i_enable_ir: true
      i_calibration_dump: true
      i_nn_type: none
      i_pipeline_type: Depth

    left:
      i_resolution: 800P
      i_add_exposure_offset: false
      i_publish_topic: true
      i_height: 800
      i_fps: 90.0
      i_board_socket_id: 1
      i_calibration_file: 'file:///home/shivam157/dai_ws/src/depthai-ros/depthai_ros_driver/config/calibration/left.yaml'

    rgb:
      i_resolution: 1080P
      i_disable_node: true
      i_add_exposure_offset: false
      i_height: 1080
      i_board_socket_id: 0

    right:
      i_resolution: 800P
      i_add_exposure_offset: false
      i_publish_topic: true
      i_height: 800
      i_fps: 90.0
      i_board_socket_id: 2
      i_calibration_file: 'file:///home/shivam157/dai_ws/src/depthai-ros/depthai_ros_driver/config/calibration/right.yaml'
      

    stereo:
      i_set_input_size: true
      i_height: 800
      i_width: 1280
      i_board_socket_id: 2
      i_reverse_stereo_socket_order: true
      i_align_depth: true
      i_publish_right_rect: true
      i_publish_left_rect: true
      i_publish_synced_rect_pair: true
      i_left_socket_id: 1
      i_right_socket_id: 2

Edit 1:
Setting i_usb_speed: SUPER_PLUS starts the images with high fps but it keeps on decreasing once started. Why is the fps decreasing. Also, how to get low Latency?

I am following this link but still not able to figure out why fps is decreasing.

Edit 2:
I am using jetson orin nano, Oak FFC 3P, two OV9282 and the USB C wire that came with the OAK module. I have powered OAK FFC 3P with the adapter it came with. Power mode is 15W in jetson.

Edit 3:
The following is when I don't run ROS driver:

shivam157@ubuntu:~/dai_ws$ python3 oak_bandwidth_test.py 
Downlink 2093.5 mbps
Uplink 1992.0 mbps
Press any key to continue...
shivam157@ubuntu:~/dai_ws$ python3 oak_latency_test.py 
Sending buffer 0
Sending buffer 1
Got 1. buffer, latency 0.83 ms
Sending buffer 2
Got 2. buffer, latency 0.89 ms
Sending buffer 3
Got 3. buffer, latency 0.97 ms
Sending buffer 4
Got 4. buffer, latency 1.07 ms
Sending buffer 5

jakaskerl

Hi @ShivamSharma
Have you tried doing the same outside ROS? Just trying to figure out whether ROS is the issue for high latency. Or perhaps if there is a trick in RVIZ preventing the full FPS? cc @Luxonis-Adam

Thanks,
Jaka

ShivamSharma

jakaskerl

In stereo mode in ROS 2, this issue is in both stereo and the depth mode pipeline of ROS 2. I can also see that the stereo/image_raw topic, which should be the depth or disparity image, is very poor. I suspect that the poor disparity has to do something with the fps because Stereo mode is working fine.

The fps is also okay when I run the cameras outside of ros. Below is a screenshot of disparity in Depth mode in ROS 2.

Also, as discussed in other discussion thread, switching the ports didn't help the disparity quality in ROS 2 but out of ROS the disparity is good. Can poor projection matrix affect the quality of disparity a lot? Since calibration does not provide projection matrix I might have made some mistake in calculation of projection matrix.

Also, even after settingi_usb_speed: SUPER_PLUS I see the following output in the terminal:
[component_container-1] [INFO] [1718380825.270871299] [oak]: USB SPEED: SUPER

Edit 1:
I changed the hardcoded baseline in the ROS 2 driver to 14 cm and then tried depth pipeline with both sockets switched and not switched. Both time the results were the same like the image I shared above. In extrinsic calibration I got 14 cm as the baseline. And I also did colcon build after I changed the hardcoded value.

ShivamSharma

@jakaskerl @Luxonis-Adam Can this issue be solved? The fps drop and unsycnhronization were also there in the stereo node.

Luxonis-Adam

Hi, could you provide output with just setting stereo.i_publish_synced_pair: true and disabling individual synced pair topics (stereo.i_publish_right_rect: false, stereo.i_publish_left_rect: false)

ShivamSharma

Luxonis-Adam

First we will have to solve the following issue:
luxonis/depthai-ros328

ShivamSharma

@Luxonis-Adam See the following:

shivam157@ubuntu:~/dai_ws_non_prod$ ros2 launch depthai_ros_driver camera.launch.py 
[INFO] [launch]: All log files can be found below /home/shivam157/.ros/log/2024-08-07-15-46-36-258635-ubuntu-28520
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container-1]: process started with pid [28534]
[component_container-1] [INFO] [1723059996.922322411] [oak_container]: Load Library: /opt/ros/humble/lib/librobot_state_publisher_node.so
[component_container-1] [INFO] [1723059996.928042442] [oak_container]: Found class: rclcpp_components::NodeFactoryTemplate<robot_state_publisher::RobotStatePublisher>
[component_container-1] [INFO] [1723059996.928128012] [oak_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<robot_state_publisher::RobotStatePublisher>
[component_container-1] [INFO] [1723059996.943975588] [oak_state_publisher]: got segment oak
[component_container-1] [INFO] [1723059996.944083751] [oak_state_publisher]: got segment oak-d-base-frame
[component_container-1] [INFO] [1723059996.944114376] [oak_state_publisher]: got segment oak_imu_frame
[component_container-1] [INFO] [1723059996.944135400] [oak_state_publisher]: got segment oak_left_camera_frame
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/oak_state_publisher' in container 'oak_container'
[component_container-1] [INFO] [1723059996.944153513] [oak_state_publisher]: got segment oak_left_camera_optical_frame
[component_container-1] [INFO] [1723059996.944169801] [oak_state_publisher]: got segment oak_model_origin
[component_container-1] [INFO] [1723059996.944184810] [oak_state_publisher]: got segment oak_rgb_camera_frame
[component_container-1] [INFO] [1723059996.944200202] [oak_state_publisher]: got segment oak_rgb_camera_optical_frame
[component_container-1] [INFO] [1723059996.944215722] [oak_state_publisher]: got segment oak_right_camera_frame
[component_container-1] [INFO] [1723059996.944229963] [oak_state_publisher]: got segment oak_right_camera_optical_frame
[component_container-1] [INFO] [1723059996.948866251] [oak_container]: Load Library: /home/shivam157/dai_ws_non_prod/install/depthai_ros_driver/lib/libdepthai_ros_driver.so
[component_container-1] [INFO] [1723059997.125719891] [oak_container]: Found class: rclcpp_components::NodeFactoryTemplate<depthai_ros_driver::Camera>
[component_container-1] [INFO] [1723059997.125874647] [oak_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<depthai_ros_driver::Camera>
[component_container-1] [INFO] [1723059997.142654056] [oak]: No ip/mxid specified, connecting to the next available device.
[component_container-1] [INFO] [1723060001.729589925] [oak]: Camera with MXID: 14442C10E14FCBD600 and Name: 1.2.1 connected!
[component_container-1] [INFO] [1723060001.730851688] [oak]: USB SPEED: SUPER
[component_container-1] [INFO] [1723060001.774894959] [oak]: Device type: OAK-FFC-3P
[component_container-1] [INFO] [1723060001.779530256] [oak]: Pipeline type: Depth
[component_container-1] [INFO] [1723060002.160754024] [oak]: Finished setting up pipeline.
[component_container-1] [INFO] [1723060002.163625047] [oak]: Saving pipeline schema to: /tmp/14442C10E14FCBD600_pipeline.json
[component_container-1] [INFO] [1723060002.228141111] [oak]: Saving calibration to: /tmp/14442C10E14FCBD600_calibration.json
[component_container-1] [INFO] [1723060003.200960062] [oak]: Setting up stereo pair sync timer with period 11 ms based on left sensor FPS.
[component_container-1] [INFO] [1723060003.922228287] [oak]: Camera ready!
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/oak' in container '/oak_container'
[component_container-1] [WARN] [1723060874.213330711] [oak]: Left and right rectified frames are not synchronized!
[component_container-1] [WARN] [1723060885.286067718] [oak]: Left and right rectified frames are not synchronized!
[component_container-1] [WARN] [1723060898.257661062] [oak]: Left and right rectified frames are not synchronized!
[component_container-1] [WARN] [1723060908.323447090] [oak]: Left and right rectified frames are not synchronized!

I will make the changes to camera.yaml and let you know the result

Edit 1:
Same result after removing (stereo.i_publish_right_rect: false, stereo.i_publish_left_rect: false).

I don't have to build again after I change the camera.yaml, right?

camera.yaml:

/oak:
  ros__parameters:
    camera:
      i_enable_imu: true
      i_enable_ir: true
      i_calibration_dump: true
      i_pipeline_dump: true
      i_nn_type: none
      i_pipeline_type: Depth
      i_usb_speed: SUPER_PLUS

    left:
      i_resolution: 800P
      i_add_exposure_offset: false
      i_publish_topic: true
      i_height: 800
      i_fps: 90.0
      i_board_socket_id: 2
      # i_calibration_file: 'file:///home/shivam157/dai_ws/src/depthai-ros/depthai_ros_driver/config/calibration/left.yaml'
      
    # rgb:
    #   i_resolution: 1080P
    #   i_disable_node: true
    #   i_add_exposure_offset: false
    #   i_height: 1080
    #   i_board_socket_id: 0

    right:
      i_resolution: 800P
      i_add_exposure_offset: false
      i_publish_topic: true
      i_height: 800
      i_fps: 90.0
      i_board_socket_id: 1
      # i_calibration_file: 'file:///home/shivam157/dai_ws/src/depthai-ros/depthai_ros_driver/config/calibration/right.yaml'
      

      

    stereo:
      i_set_input_size: true
      i_height: 800
      i_width: 1280
      i_board_socket_id: 2
      i_reverse_stereo_socket_order: false
      i_align_depth: true
      #i_publish_right_rect: true
      #i_publish_left_rect: true
      i_publish_synced_rect_pair: true
      i_left_socket_id: 1
      i_right_socket_id: 2

Luxonis-Adam

Hi, I think several things could come into factor here:

Large size of stereo depth at large frequency - I would try reducing the resolution to 400P and remove all width/height setting
When checking frequency with ros2 topic hz there is an additional overhead since those messages must be deserialized via ROS2 Python libraries which are known for their issues with the performance, there might be also additional overhead from the Jetson itself
You can use ROS2 mechanisms such as Composition/Intra process communication to mitigate some of the overhead coming from DDS mechanism. You can also tune DDS for larger messages (all of this information can be found in ROS2 documentation, for example https://docs.ros.org/en/jazzy/How-To-Guides/DDS-tuning.html)
Using low_bandwidth mode could help with the performance as it is sending encoded data