How to synchronize rectified left and right frames ?

Mmichaelphlin · Jul 29, 2024

Hi @erik , I have an OAK-D Pro and NVIDIA AGX Orin.
Can I run VIO with `dai.node.RTABMapVIO` in this setup? or does it only support RCV3?
and is there any documentation associated to VIO or VSLAM?
Thanks for you feedback in advance.

Michael

erik · Jul 29, 2024

michaelphlin Yep you can, in that case the RTABMapVIO would run on the NVIDIA AGX Orin, so device would only provide color/depth/IMU stream.

ShivamSharma · Jul 29, 2024

erik

I ran Python3 install_requirements.py today and I still see that it is building the wheels and giving an error:

shivam157@ubuntu:~/depthai-core/examples/python$ python3 install_requirements.py
pip 24.2 from /home/shivam157/.local/lib/python3.10/site-packages/pip (python 3.10)
Requirement already satisfied: pip in /home/shivam157/.local/lib/python3.10/site-packages (24.2)
Requirement already satisfied: pyyaml in /home/shivam157/.local/lib/python3.10/site-packages (6.0.1)
Requirement already satisfied: requests in /home/shivam157/.local/lib/python3.10/site-packages (2.32.3)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/shivam157/.local/lib/python3.10/site-packages (from requests) (2.0.12)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests) (3.3)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/lib/python3/dist-packages (from requests) (1.26.5)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests) (2020.6.20)
Looking in indexes: https://pypi.org/simple, https://artifacts.luxonis.com/artifactory/luxonis-python-snapshot-local
Collecting depthai==3.0.0-alpha.0.dev0+25a9f0057d769b430bf3a543eaa8f33254518d03
  Using cached https://artifacts.luxonis.com/artifactory/luxonis-python-snapshot-local/depthai/depthai-3.0.0a0.dev0%2B25a9f0057d769b430bf3a543eaa8f33254518d03.tar.gz (187 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: depthai
  Building wheel for depthai (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for depthai (pyproject.toml) did not run successfully.
  \u2502 exit code: 1
  \u2570\u2500> [68 lines of output]
      /tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/dist.py:289: InformationOnly: Normalizing '3.0.0-alpha.0.dev0+25a9f0057d769b430bf3a543eaa8f33254518d03' to '3.0.0a0.dev0+25a9f0057d769b430bf3a543eaa8f33254518d03'
        self.metadata.version = self._normalize_version(self.metadata.version)
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-aarch64-cpython-310
      creating build/lib.linux-aarch64-cpython-310/depthai_cli
      copying depthai_cli/__init__.py -> build/lib.linux-aarch64-cpython-310/depthai_cli
      copying depthai_cli/depthai_cli.py -> build/lib.linux-aarch64-cpython-310/depthai_cli
      running build_ext
      CMake Warning:
        Ignoring extra path from command line:
      
         "/tmp"
      
      
      CMake Error: The source directory "/tmp" does not appear to contain CMakeLists.txt.
      Specify --help for usage, or press the help button on the CMake GUI.
      Traceback (most recent call last):
        File "/home/shivam157/.local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/shivam157/.local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/shivam157/.local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 415, in build_wheel
          return self._build_with_temp_dir(
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 397, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 497, in run_setup
          super().run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 313, in run_setup
          exec(code, locals())
        File "<string>", line 220, in <module>
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/__init__.py", line 108, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 184, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 200, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 970, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/dist.py", line 945, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 989, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/command/bdist_wheel.py", line 373, in run
          self.run_command("build")
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/dist.py", line 945, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 989, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/dist.py", line 945, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-8tz847ck/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 989, in run_command
          cmd_obj.run()
        File "<string>", line 93, in run
        File "<string>", line 217, in build_extension
        File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/tmp', '-DDEPTHAI_BUILD_PYTHON=ON', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/pip-install-sjf68bin/depthai_306aec13e4264ef3b83fdb9d5d84ad54/build/lib.linux-aarch64-cpython-310/', '-DPYTHON_EXECUTABLE=/usr/bin/python3', '-DDEPTHAI_PYTHON_COMMIT_HASH=25a9f0057d769b430bf3a543eaa8f33254518d03', '-DCMAKE_BUILD_TYPE=Release', '-DHUNTER_CONFIGURATION_TYPES=Release', '-DHUNTER_JOBS_NUMBER=2']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for depthai
Failed to build depthai
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (depthai)
Processing /home/shivam157/depthai-core/bindings/python
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: depthai
  Building wheel for depthai (pyproject.toml) ... /


  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for depthai
Failed to build depthai
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (depthai)
Couldn't install dependencies as wheels and trying to compile from sources failed

Mmichaelphlin · Jul 30, 2024

ShivamSharma

@erik I have the same issue today after I follow the CMake instruction with v3_develop branch.

Is there any instruction can guide us step by step setup this VIO/ Rtabmap SLAM pipeline?

Thanks.

Michael

erik · Jul 30, 2024

@michaelphlin @ShivamSharma
Checkout to 4b6fb2bd on depthai-core. I used this commit as it has already built wheels. Go to examples/python, create venv, source it, call install_requirements.py. This will install required depthai libraries. Then go to RVC2/VSLAM folder, and run eg. basalt_vio.py. Note that this is heavily WIP, so some things won't work.

Mmichaelphlin · Jul 30, 2024

@erik Thanks for your help!!! the lib dependency is not longer a issue now.
After adding imu.enableFirmwareUpdate(True) line into rtabmap_vio_slam.py and re-running it, I receive the following error message:
Does that means I need to calibrate the IMU and camera again? or is there any factor calibrated matrix I can use for SLAM?
python3 rtabmap_vio_slam.py

[1844301041693CF500] [3.6] [1.682] [IMU(2)] [warning] IMU 'enableFirmwareUpdate' is deprecated! Use 'device.startIMUFirmwareUpdate'

[1844301041693CF500] [3.6] [1.710] [IMU(2)] [warning] Performing IMU firmware update, do NOT disconnect the device.

[1844301041693CF500] [3.6] [1.832] [IMU(2)] [warning] IMU firmware update status: 0%

[2024-07-30 15:59:45.717] [ThreadedNode] [error] Node threw exception, stopping the node. Exception message: IMU calibration data is not available on device yet.

erik · Jul 30, 2024

@michaelphlin are you using a device that has BNO085/6 imu?

Mmichaelphlin · Jul 30, 2024

Hi @erik

Yes, based on the spec, I should have BNO086 on my OAK-D Pro and OAK-D Pro W.

If there’s extra step I need to initialize my IMU, please kindly let me know.

I really appreciate your help on this issue.

sincerely,

Michael

jakaskerl · Jul 31, 2024

Hi @michaelphlin
You can try adding the IMU extrinsics to the calibration of the device: luxonis/depthai-boardsblob/adding_IMU_tmp/boards/OAK-D-PRO.json

LMK if it solves the issue.

Thanks,
Jaka

ShivamSharma · Aug 3, 2024

@jakaskerl @erik Does depthai API or SDK have a script to calibrate IMU? How do I calculate the IMU extrinsics of the OAK FFC 3P? Will the slam work without IMU?

erik · Aug 5, 2024

Hi @ShivamSharma ,
We do not, extrinsics have to be extracted from CAD model (or measured). It needs to be spec-ed in the json file like this:
luxonis/depthai-boardsblob/main/boards/OAK-D-CM4-POE.json#L52-L72

ShivamSharma · Aug 5, 2024

erik So can I use the depthai slam without an imu?

erik · Aug 5, 2024

@ShivamSharma I don't think so, this is very much in WIP so we haven't done many tests/evaluations.