• DepthAIROS

  • How to synchronize rectified left and right frames ?

erik
According to SLAM algorithms like RTABMAP and Isaac SLAM, it doesn't synchronize. Can you help me get good output from SLAM? I think the current issue is synchronization. I get jitter messages of 66 ms or 100 ms difference between consecutive frames when I run Isaac slam, and in RTABMAP I get a constant 0.033 s difference between the left and right frames. Rtabmap also complains sometimes that it has not received frames in the 5-second timeframe, even though the fps is around 25. I thought this was related to QoS policy in ROS, but I have tried different policies and have had no success with SLAM so far.

The QoS policy is the following:

// Create a custom QoS profile
    rmw_qos_profile_t custom_qos_profile = rmw_qos_profile_default;
    custom_qos_profile.history = RMW_QOS_POLICY_HISTORY_KEEP_LAST;
    custom_qos_profile.depth = 1; // Keep only the last message
    custom_qos_profile.reliability = RMW_QOS_POLICY_RELIABILITY_BEST_EFFORT;
    custom_qos_profile.durability = RMW_QOS_POLICY_DURABILITY_VOLATILE;
    custom_qos_profile.deadline = RMW_QOS_DEADLINE_DEFAULT;
    custom_qos_profile.lifespan = RMW_QOS_LIFESPAN_DEFAULT;
    custom_qos_profile.liveliness = RMW_QOS_POLICY_LIVELINESS_SYSTEM_DEFAULT;
    custom_qos_profile.liveliness_lease_duration =
        RMW_QOS_LIVELINESS_LEASE_DURATION_DEFAULT;
    custom_qos_profile.avoid_ros_namespace_conventions = false;

    // Convert rmw_qos_profile_t to rclcpp::QoS
    auto qos =
        rclcpp::QoS(rclcpp::QoSInitialization::from_rmw(custom_qos_profile));

    rectifLeftImagePub = this->create_publisher<sensor_msgs::msg::Image>(
        "left/image_rect", qos);

I process the synchronized raw frames with OpenCV using the rectification map created from the calibration, and then publish them next to each other inside the while loop. Where am I going wrong? I give the Depthai timestamp to the header of info and image messages.

Like the following:

rclcpp::Time frame_time(
            leftFrame->getTimestamp().time_since_epoch().count());

If you could help me with this issue, I would really appreciate it. My employment depends on getting the slam working on this OAK camera setup.

    erik
    Thanks a lot for your quick replies.

    Will the slam that you mentioned run on the Jetson Orin Nano in real time with the custom oak camera setup I have?

    Which branch of depthai_core I have to switch to? I currently use the main branch, which doesn't have a folder for slam. Will I be able to get the map in ROS 2 from this slam?

    • erik replied to this.

        ShivamSharma Yes, it will. Should even run on RPI5. Branch is in the URL, v3_develop. Not sure about the ros2 part, but you can just publish the map to ros I assume.

            erik

            Hi,

            I changed the branch to the v3_develop branch. I installed it using the cmake command given in the readme file. When I first ran the slam file, I got an error for rerun and then I ran:
            $ /usr/bin/python3 /home/shivam157/depthai-core/examples/python/install_requirements.py --install_rerun

            Which gave me the following error:

            ERROR: Failed building wheel for depthai
Failed to build depthai
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (depthai)
Couldn't install dependencies as wheels and trying to compile from sources failed

            Depthai-python was installed I don't know what the issue is.

            After that, I ran:

            
$ python3 rtabmap_vio_slam.py 
Traceback (most recent call last):
  File "/home/shivam157/depthai-core/examples/python/RVC2/VSLAM/rtabmap_vio_slam.py", line 3, in <module>
    from rerun_node import RerunNode
  File "/home/shivam157/depthai-core/examples/python/RVC2/VSLAM/rerun_node.py", line 13, in <module>
    class RerunNode(dai.node.ThreadedHostNode):
AttributeError: module 'depthai.node' has no attribute 'ThreadedHostNode'

            What do I need to do to run this SLAM?

              erik
              I have never used Github Actions, so it is not very clear to me how I can install Python Depthai. Can't I just use pip3 install depthai? Like given in this website

              How long do I have to wait for wheels to be ready?

                Hi @erik , I have an OAK-D Pro and NVIDIA AGX Orin.
                Can I run VIO with `dai.node.RTABMapVIO` in this setup? or does it only support RCV3?
                and is there any documentation associated to VIO or VSLAM?
                Thanks for you feedback in advance.

                Michael

                • erik replied to this.

                  michaelphlin Yep you can, in that case the RTABMapVIO would run on the NVIDIA AGX Orin, so device would only provide color/depth/IMU stream.

                    erik

                    I ran Python3 install_requirements.py today and I still see that it is building the wheels and giving an error:

                    shivam157@ubuntu:~/depthai-core/examples/python$ python3 install_requirements.py
pip 24.2 from /home/shivam157/.local/lib/python3.10/site-packages/pip (python 3.10)
Requirement already satisfied: pip in /home/shivam157/.local/lib/python3.10/site-packages (24.2)
Requirement already satisfied: pyyaml in /home/shivam157/.local/lib/python3.10/site-packages (6.0.1)
Requirement already satisfied: requests in /home/shivam157/.local/lib/python3.10/site-packages (2.32.3)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/shivam157/.local/lib/python3.10/site-packages (from requests) (2.0.12)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests) (3.3)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/lib/python3/dist-packages (from requests) (1.26.5)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests) (2020.6.20)
Looking in indexes: https://pypi.org/simple, https://artifacts.luxonis.com/artifactory/luxonis-python-snapshot-local
Collecting depthai==3.0.0-alpha.0.dev0+25a9f0057d769b430bf3a543eaa8f33254518d03
  Using cached https://artifacts.luxonis.com/artifactory/luxonis-python-snapshot-local/depthai/depthai-3.0.0a0.dev0%2B25a9f0057d769b430bf3a543eaa8f33254518d03.tar.gz (187 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: depthai
  Building wheel for depthai (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for depthai (pyproject.toml) did not run successfully.
  \u2502 exit code: 1
  \u2570\u2500> [68 lines of output]
      /tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/dist.py:289: InformationOnly: Normalizing '3.0.0-alpha.0.dev0+25a9f0057d769b430bf3a543eaa8f33254518d03' to '3.0.0a0.dev0+25a9f0057d769b430bf3a543eaa8f33254518d03'
        self.metadata.version = self._normalize_version(self.metadata.version)
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-aarch64-cpython-310
      creating build/lib.linux-aarch64-cpython-310/depthai_cli
      copying depthai_cli/__init__.py -> build/lib.linux-aarch64-cpython-310/depthai_cli
      copying depthai_cli/depthai_cli.py -> build/lib.linux-aarch64-cpython-310/depthai_cli
      running build_ext
      CMake Warning:
        Ignoring extra path from command line:
      
         "/tmp"
      
      
      CMake Error: The source directory "/tmp" does not appear to contain CMakeLists.txt.
      Specify --help for usage, or press the help button on the CMake GUI.
      Traceback (most recent call last):
        File "/home/shivam157/.local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/shivam157/.local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/shivam157/.local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 415, in build_wheel
          return self._build_with_temp_dir(
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 397, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 497, in run_setup
          super().run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 313, in run_setup
          exec(code, locals())
        File "<string>", line 220, in <module>
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/__init__.py", line 108, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 184, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 200, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 970, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/dist.py", line 945, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 989, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/command/bdist_wheel.py", line 373, in run
          self.run_command("build")
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/dist.py", line 945, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 989, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/dist.py", line 945, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 989, in run_command
          cmd_obj.run()
        File "<string>", line 93, in run
        File "<string>", line 217, in build_extension
        File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/tmp', '-DDEPTHAI_BUILD_PYTHON=ON', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/pip-install-sjf68bin/depthai_306aec13e4264ef3b83fdb9d5d84ad54/build/lib.linux-aarch64-cpython-310/', '-DPYTHON_EXECUTABLE=/usr/bin/python3', '-DDEPTHAI_PYTHON_COMMIT_HASH=25a9f0057d769b430bf3a543eaa8f33254518d03', '-DCMAKE_BUILD_TYPE=Release', '-DHUNTER_CONFIGURATION_TYPES=Release', '-DHUNTER_JOBS_NUMBER=2']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for depthai
Failed to build depthai
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (depthai)
Processing /home/shivam157/depthai-core/bindings/python
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: depthai
  Building wheel for depthai (pyproject.toml) ... /


  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for depthai
Failed to build depthai
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (depthai)
Couldn't install dependencies as wheels and trying to compile from sources failed

                        ShivamSharma

                        @erik I have the same issue today after I follow the CMake instruction with v3_develop branch.

                        Is there any instruction can guide us step by step setup this VIO/ Rtabmap SLAM pipeline?

                        Thanks.

                        Michael

                          @michaelphlin @ShivamSharma
                          Checkout to 4b6fb2bd on depthai-core. I used this commit as it has already built wheels. Go to examples/python, create venv, source it, call install_requirements.py. This will install required depthai libraries. Then go to RVC2/VSLAM folder, and run eg. basalt_vio.py. Note that this is heavily WIP, so some things won't work.

                          @erik Thanks for your help!!! the lib dependency is not longer a issue now.
                          After adding imu.enableFirmwareUpdate(True) line into rtabmap_vio_slam.py and re-running it, I receive the following error message:
                          Does that means I need to calibrate the IMU and camera again? or is there any factor calibrated matrix I can use for SLAM?
                          python3 rtabmap_vio_slam.py

                          [1844301041693CF500] [3.6] [1.682] [IMU(2)] [warning] IMU 'enableFirmwareUpdate' is deprecated! Use 'device.startIMUFirmwareUpdate'

                          [1844301041693CF500] [3.6] [1.710] [IMU(2)] [warning] Performing IMU firmware update, do NOT disconnect the device.

                          [1844301041693CF500] [3.6] [1.832] [IMU(2)] [warning] IMU firmware update status: 0%

                          [2024-07-30 15:59:45.717] [ThreadedNode] [error] Node threw exception, stopping the node. Exception message: IMU calibration data is not available on device yet.

                          Hi @erik

                          Yes, based on the spec, I should have BNO086 on my OAK-D Pro and OAK-D Pro W.

                          If there’s extra step I need to initialize my IMU, please kindly let me know.

                          I really appreciate your help on this issue.

                          sincerely,

                          Michael

                          @jakaskerl @erik Does depthai API or SDK have a script to calibrate IMU? How do I calculate the IMU extrinsics of the OAK FFC 3P? Will the slam work without IMU?