How to synchronize rectified left and right frames ?

erik
According to SLAM algorithms like RTABMAP and Isaac SLAM, it doesn't synchronize. Can you help me get good output from SLAM? I think the current issue is synchronization. I get jitter messages of 66 ms or 100 ms difference between consecutive frames when I run Isaac slam, and in RTABMAP I get a constant 0.033 s difference between the left and right frames. Rtabmap also complains sometimes that it has not received frames in the 5-second timeframe, even though the fps is around 25. I thought this was related to QoS policy in ROS, but I have tried different policies and have had no success with SLAM so far.

The QoS policy is the following:

// Create a custom QoS profile
    rmw_qos_profile_t custom_qos_profile = rmw_qos_profile_default;
    custom_qos_profile.history = RMW_QOS_POLICY_HISTORY_KEEP_LAST;
    custom_qos_profile.depth = 1; // Keep only the last message
    custom_qos_profile.reliability = RMW_QOS_POLICY_RELIABILITY_BEST_EFFORT;
    custom_qos_profile.durability = RMW_QOS_POLICY_DURABILITY_VOLATILE;
    custom_qos_profile.deadline = RMW_QOS_DEADLINE_DEFAULT;
    custom_qos_profile.lifespan = RMW_QOS_LIFESPAN_DEFAULT;
    custom_qos_profile.liveliness = RMW_QOS_POLICY_LIVELINESS_SYSTEM_DEFAULT;
    custom_qos_profile.liveliness_lease_duration =
        RMW_QOS_LIVELINESS_LEASE_DURATION_DEFAULT;
    custom_qos_profile.avoid_ros_namespace_conventions = false;

    // Convert rmw_qos_profile_t to rclcpp::QoS
    auto qos =
        rclcpp::QoS(rclcpp::QoSInitialization::from_rmw(custom_qos_profile));

    rectifLeftImagePub = this->create_publisher<sensor_msgs::msg::Image>(
        "left/image_rect", qos);

I process the synchronized raw frames with OpenCV using the rectification map created from the calibration, and then publish them next to each other inside the while loop. Where am I going wrong? I give the Depthai timestamp to the header of info and image messages.

Like the following:

rclcpp::Time frame_time(
            leftFrame->getTimestamp().time_since_epoch().count());

If you could help me with this issue, I would really appreciate it. My employment depends on getting the slam working on this OAK camera setup.

erik
Thanks a lot for your quick replies.

Will the slam that you mentioned run on the Jetson Orin Nano in real time with the custom oak camera setup I have?

Which branch of depthai_core I have to switch to? I currently use the main branch, which doesn't have a folder for slam. Will I be able to get the map in ROS 2 from this slam?

erik

Hi,

I changed the branch to the v3_develop branch. I installed it using the cmake command given in the readme file. When I first ran the slam file, I got an error for rerun and then I ran:
$ /usr/bin/python3 /home/shivam157/depthai-core/examples/python/install_requirements.py --install_rerun

Which gave me the following error:

ERROR: Failed building wheel for depthai
Failed to build depthai
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (depthai)
Couldn't install dependencies as wheels and trying to compile from sources failed

Depthai-python was installed I don't know what the issue is.

After that, I ran:

$ python3 rtabmap_vio_slam.py 
Traceback (most recent call last):
  File "/home/shivam157/depthai-core/examples/python/RVC2/VSLAM/rtabmap_vio_slam.py", line 3, in <module>
    from rerun_node import RerunNode
  File "/home/shivam157/depthai-core/examples/python/RVC2/VSLAM/rerun_node.py", line 13, in <module>
    class RerunNode(dai.node.ThreadedHostNode):
AttributeError: module 'depthai.node' has no attribute 'ThreadedHostNode'

What do I need to do to run this SLAM?

erik

I ran Python3 install_requirements.py today and I still see that it is building the wheels and giving an error:

shivam157@ubuntu:~/depthai-core/examples/python$ python3 install_requirements.py
pip 24.2 from /home/shivam157/.local/lib/python3.10/site-packages/pip (python 3.10)
Requirement already satisfied: pip in /home/shivam157/.local/lib/python3.10/site-packages (24.2)
Requirement already satisfied: pyyaml in /home/shivam157/.local/lib/python3.10/site-packages (6.0.1)
Requirement already satisfied: requests in /home/shivam157/.local/lib/python3.10/site-packages (2.32.3)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/shivam157/.local/lib/python3.10/site-packages (from requests) (2.0.12)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests) (3.3)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/lib/python3/dist-packages (from requests) (1.26.5)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests) (2020.6.20)
Looking in indexes: https://pypi.org/simple, https://artifacts.luxonis.com/artifactory/luxonis-python-snapshot-local
Collecting depthai==3.0.0-alpha.0.dev0+25a9f0057d769b430bf3a543eaa8f33254518d03
  Using cached https://artifacts.luxonis.com/artifactory/luxonis-python-snapshot-local/depthai/depthai-3.0.0a0.dev0%2B25a9f0057d769b430bf3a543eaa8f33254518d03.tar.gz (187 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: depthai
  Building wheel for depthai (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for depthai (pyproject.toml) did not run successfully.
  \u2502 exit code: 1
  \u2570\u2500> [68 lines of output]
      /tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/dist.py:289: InformationOnly: Normalizing '3.0.0-alpha.0.dev0+25a9f0057d769b430bf3a543eaa8f33254518d03' to '3.0.0a0.dev0+25a9f0057d769b430bf3a543eaa8f33254518d03'
        self.metadata.version = self._normalize_version(self.metadata.version)
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-aarch64-cpython-310
      creating build/lib.linux-aarch64-cpython-310/depthai_cli
      copying depthai_cli/__init__.py -> build/lib.linux-aarch64-cpython-310/depthai_cli
      copying depthai_cli/depthai_cli.py -> build/lib.linux-aarch64-cpython-310/depthai_cli
      running build_ext
      CMake Warning:
        Ignoring extra path from command line:
      
         "/tmp"
      
      
      CMake Error: The source directory "/tmp" does not appear to contain CMakeLists.txt.
      Specify --help for usage, or press the help button on the CMake GUI.
      Traceback (most recent call last):
        File "/home/shivam157/.local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/shivam157/.local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/shivam157/.local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 415, in build_wheel
          return self._build_with_temp_dir(
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 397, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 497, in run_setup
          super().run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 313, in run_setup
          exec(code, locals())
        File "<string>", line 220, in <module>
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/__init__.py", line 108, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 184, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 200, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 970, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/dist.py", line 945, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 989, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/command/bdist_wheel.py", line 373, in run
          self.run_command("build")
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/dist.py", line 945, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 989, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/dist.py", line 945, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 989, in run_command
          cmd_obj.run()
        File "<string>", line 93, in run
        File "<string>", line 217, in build_extension
        File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/tmp', '-DDEPTHAI_BUILD_PYTHON=ON', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/pip-install-sjf68bin/depthai_306aec13e4264ef3b83fdb9d5d84ad54/build/lib.linux-aarch64-cpython-310/', '-DPYTHON_EXECUTABLE=/usr/bin/python3', '-DDEPTHAI_PYTHON_COMMIT_HASH=25a9f0057d769b430bf3a543eaa8f33254518d03', '-DCMAKE_BUILD_TYPE=Release', '-DHUNTER_CONFIGURATION_TYPES=Release', '-DHUNTER_JOBS_NUMBER=2']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for depthai
Failed to build depthai
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (depthai)
Processing /home/shivam157/depthai-core/bindings/python
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: depthai
  Building wheel for depthai (pyproject.toml) ... /


  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for depthai
Failed to build depthai
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (depthai)
Couldn't install dependencies as wheels and trying to compile from sources failed

@jakaskerl @erik Does depthai API or SDK have a script to calibrate IMU? How do I calculate the IMU extrinsics of the OAK FFC 3P? Will the slam work without IMU?