Oak-D Pro active stereo quality

mdupre27

Hi,

I am trying to get some depth data using the active stereo sensor. However i am getting a lot of outliers imn most of the image. Is it to be expected ? Am I missing some kind of calibration ?

mdupre27

First image is a flat wall at 50cm. second image is a start chart at 1m (wall behind is at 1.7m).

jakaskerl

Hi mdupre27
1st looks like a bad calibration since you are getting a lot of distortion on your depth map. Recalibrating the camera using https://docs.luxonis.com/projects/hardware/en/latest/pages/guides/calibration/ should likely fix the issue. For the second picture, are you applying any post-processing filters?

Thanks,
Jaka

mdupre27

No I am not applying anything yet. So sensors are shipped without calibration ? Or do they all have a same standard calibration that is not optimal ?

jakaskerl

mdupre27
Devices have onboard factory calibration. It could be that the calibration was sub-optimal or that environment changes during shipping caused the existing calibration to no longer be valid.

Thanks,
Jaka

mdupre27

I tested multiple sensors and they all come up with similar pattern. I will try to recalibrate.

Thank you !

mdupre27

Trying to run the calibration script. Howver i dont manage to install the requirements on my windows 10, specifically build the depthai wheel produces the following error:

(luxonisenv) C:\Users\mdupr\Documents\depthai-luxonis\depthai>pip install depthai

Collecting depthai

Downloading depthai-2.24.0.0.tar.gz (434 kB)

 ---------------------------------------- 434.0/434.0 kB 9.0 MB/s eta 0:00:00

Installing build dependencies ... done

Getting requirements to build wheel ... done

Preparing metadata (pyproject.toml) ... done

Building wheels for collected packages: depthai

Building wheel for depthai (pyproject.toml) ... error

error: subprocess-exited-with-error

× Building wheel for depthai (pyproject.toml) did not run successfully.

│ exit code: 1

╰─> [88 lines of output]

  running bdist_wheel

  running build

  running build_ext

  <string>:87: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

  -- Building for: NMake Makefiles

  CMake Error at CMakeLists.txt:2 (project):

    Running

     'nmake' '-?'

    failed with:

     The system cannot find the file specified

  [hunter \*\* INTERNAL \*\*] Configure project failed.

  [hunter \*\* INTERNAL \*\*] To reproduce the error run: C:/Users/mdupr/AppData/Local/Temp/pip-build-env-lw5no1dd/overlay/Lib/site-packages/cmake/data/bin/cmake.exe -HC:/.hunter/_Base/Download/Hunter/0.23.322/cb0ea1f -BC:/.hunter/_Base/Download/Hunter/0.23.322/cb0ea1f/Build -GNMake Makefiles -DCMAKE_TOOLCHAIN_FILE=C:/Users/mdupr/AppData/Local/Temp/pip-install-5fz26pbr/depthai_16e4c6eb953f4a738b4b11eaad314c40/cmake/toolchain/msvc.cmake

  [hunter \*\* INTERNAL \*\*] In directory C:/.hunter/_Base/Download/Hunter/0.23.322/cb0ea1f

  [hunter \*\* INTERNAL \*\*] [Directory:C:/Users/mdupr/AppData/Local/Temp/pip-install-5fz26pbr/depthai_16e4c6eb953f4a738b4b11eaad314c40]

  ------------------------------ ERROR ------------------------------

      https://docs.hunter.sh/en/latest/reference/errors/error.internal.html

  -------------------------------------------------------------------

  CMake Error at cmake/HunterGate.cmake:87 (message):

  Call Stack (most recent call first):

    cmake/HunterGate.cmake:97 (hunter_gate_error_page)

    cmake/HunterGate.cmake:325 (hunter_gate_internal_error)

    cmake/HunterGate.cmake:509 (hunter_gate_download)

    CMakeLists.txt:25 (HunterGate)

  -- Configuring incomplete, errors occurred!

  Traceback (most recent call last):

    File "C:\\Users\\mdupr\\AppData\\Roaming\\Python\\Python312\\site-packages\\pip\\_vendor\\pyproject_hooks\\_in_process\\_in_process.py", line 353, in <module>

      main()

    File "C:\\Users\\mdupr\\AppData\\Roaming\\Python\\Python312\\site-packages\\pip\\_vendor\\pyproject_hooks\\_in_process\\_in_process.py", line 335, in main

      json_out['return_val'] = hook(\*\*hook_input['kwargs'])

                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

    File "C:\\Users\\mdupr\\AppData\\Roaming\\Python\\Python312\\site-packages\\pip\\_vendor\\pyproject_hooks\\_in_process\\_in_process.py", line 251, in build_wheel

      return _build_backend().build_wheel(wheel_directory, config_settings,

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

    File "C:\\Users\\mdupr\\AppData\\Local\\Temp\\pip-build-env-lw5no1dd\\overlay\\Lib\\site-packages\\setuptools\\build_meta.py", line 404, in build_wheel

      return self._build_with_temp_dir(

             ^^^^^^^^^^^^^^^^^^^^^^^^^^

    File "C:\\Users\\mdupr\\AppData\\Local\\Temp\\pip-build-env-lw5no1dd\\overlay\\Lib\\site-packages\\setuptools\\build_meta.py", line 389, in _build_with_temp_dir

      self.run_setup()

    File "C:\\Users\\mdupr\\AppData\\Local\\Temp\\pip-build-env-lw5no1dd\\overlay\\Lib\\site-packages\\setuptools\\build_meta.py", line 480, in run_setup

      super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)

    File "C:\\Users\\mdupr\\AppData\\Local\\Temp\\pip-build-env-lw5no1dd\\overlay\\Lib\\site-packages\\setuptools\\build_meta.py", line 311, in run_setup

      exec(code, locals())

    File "<string>", line 198, in <module>

    File "C:\\Users\\mdupr\\AppData\\Local\\Temp\\pip-build-env-lw5no1dd\\overlay\\Lib\\site-packages\\setuptools\\__init__.py", line 103, in setup

      return distutils.core.setup(\*\*attrs)

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

    File "C:\\Users\\mdupr\\AppData\\Local\\Temp\\pip-build-env-lw5no1dd\\overlay\\Lib\\site-packages\\setuptools\\_distutils\\core.py", line 185, in setup

      return run_commands(dist)

             ^^^^^^^^^^^^^^^^^^

    File "C:\\Users\\mdupr\\AppData\\Local\\Temp\\pip-build-env-lw5no1dd\\overlay\\Lib\\site-packages\\setuptools\\_distutils\\core.py", line 201, in run_commands

      dist.run_commands()

    File "C:\\Users\\mdupr\\AppData\\Local\\Temp\\pip-build-env-lw5no1dd\\overlay\\Lib\\site-packages\\setuptools\\_distutils\\dist.py", line 969, in run_commands

      self.run_command(cmd)

    File "C:\\Users\\mdupr\\AppData\\Local\\Temp\\pip-build-env-lw5no1dd\\overlay\\Lib\\site-packages\\setuptools\\dist.py", line 963, in run_command

      super().run_command(command)

    File "C:\\Users\\mdupr\\AppData\\Local\\Temp\\pip-build-env-lw5no1dd\\overlay\\Lib\\site-packages\\setuptools\\_distutils\\dist.py", line 988, in run_command

      cmd_obj.run()

    File "C:\\Users\\mdupr\\AppData\\Local\\Temp\\pip-build-env-lw5no1dd\\overlay\\Lib\\site-packages\\wheel\\bdist_wheel.py", line 368, in run

      self.run_command("build")

    File "C:\\Users\\mdupr\\AppData\\Local\\Temp\\pip-build-env-lw5no1dd\\overlay\\Lib\\site-packages\\setuptools\\_distutils\\cmd.py", line 318, in run_command

      self.distribution.run_command(command)

    File "C:\\Users\\mdupr\\AppData\\Local\\Temp\\pip-build-env-lw5no1dd\\overlay\\Lib\\site-packages\\setuptools\\dist.py", line 963, in run_command

      super().run_command(command)

    File "C:\\Users\\mdupr\\AppData\\Local\\Temp\\pip-build-env-lw5no1dd\\overlay\\Lib\\site-packages\\setuptools\\_distutils\\dist.py", line 988, in run_command

      cmd_obj.run()

    File "C:\\Users\\mdupr\\AppData\\Local\\Temp\\pip-build-env-lw5no1dd\\overlay\\Lib\\site-packages\\setuptools\\_distutils\\command\\build.py", line 131, in run

      self.run_command(cmd_name)

    File "C:\\Users\\mdupr\\AppData\\Local\\Temp\\pip-build-env-lw5no1dd\\overlay\\Lib\\site-packages\\setuptools\\_distutils\\cmd.py", line 318, in run_command

      self.distribution.run_command(command)

    File "C:\\Users\\mdupr\\AppData\\Local\\Temp\\pip-build-env-lw5no1dd\\overlay\\Lib\\site-packages\\setuptools\\dist.py", line 963, in run_command

      super().run_command(command)

    File "C:\\Users\\mdupr\\AppData\\Local\\Temp\\pip-build-env-lw5no1dd\\overlay\\Lib\\site-packages\\setuptools\\_distutils\\dist.py", line 988, in run_command

      cmd_obj.run()

    File "<string>", line 92, in run

    File "<string>", line 195, in build_extension

    File "C:\\Users\\mdupr\\.conda\\envs\\luxonisenv\\Lib\\subprocess.py", line 413, in check_call

      raise CalledProcessError(retcode, cmd)

  subprocess.CalledProcessError: Command '['cmake', 'C:\\\\Users\\\\mdupr\\\\AppData\\\\Local\\\\Temp\\\\pip-install-5fz26pbr\\\\depthai_16e4c6eb953f4a738b4b11eaad314c40', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=C:\\\\Users\\\\mdupr\\\\AppData\\\\Local\\\\Temp\\\\pip-install-5fz26pbr\\\\depthai_16e4c6eb953f4a738b4b11eaad314c40\\\\build\\\\lib.win-amd64-cpython-312\\\\', '-DPYTHON_EXECUTABLE=C:\\\\Users\\\\mdupr\\\\.conda\\\\envs\\\\luxonisenv\\\\python.exe', '-DCMAKE_BUILD_TYPE=Release', '-DHUNTER_CONFIGURATION_TYPES=Release', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY_RELEASE=C:\\\\Users\\\\mdupr\\\\AppData\\\\Local\\\\Temp\\\\pip-install-5fz26pbr\\\\depthai_16e4c6eb953f4a738b4b11eaad314c40\\\\build\\\\lib.win-amd64-cpython-312\\\\', '-DCMAKE_TOOLCHAIN_FILE=C:\\\\Users\\\\mdupr\\\\AppData\\\\Local\\\\Temp\\\\pip-install-5fz26pbr\\\\depthai_16e4c6eb953f4a738b4b11eaad314c40/cmake/toolchain/msvc.cmake', '-A', 'x64']' returned non-zero exit status 1.

  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

ERROR: Failed building wheel for depthai

Failed to build depthai

ERROR: Could not build wheels for depthai, which is required to install pyproject.toml-based projects

jakaskerl

Hi @mdupre27
Are you by any chance using python3.12?

https://discuss.luxonis.com/d/3360-error-could-not-build-wheels-for-depthai-pyprojecttoml-based-projects/2
Installing Microsoft Visual Studio and build tools might also help with the NMAKE issue.

Thanks,
Jaka

mdupre27

Indeed, it is the version intalled by default with conda. I could not find which version to use in the requirements.

mdupre27

Installation succeeds now with python 3.11, thank you.

When I am trying to run the script, i get this error now:

I also get an issue when i try to run the simple demo:

In the GUI the "depth" is greyed out.

the camera seems to be recognized in the device manager:

but sometimes it shows up differently as Movidius MyriadX:

jakaskerl

Hi mdupre27
looks like you are using depthai==2.21.2.0. Please upgrade to the latest.

Thanks,
Jaka

mdupre27

I ran the script python install_requirements.py that installs the requirements.txt. It would be nice to have an up to date documentation. Please let me know which version to install for each package. Current requirements.txt contains:

requests==2.26.0

--extra-index-url https://www.piwheels.org/simple

numpy>=1.21.4 # For RPi Buster (last successful build) and macOS M1 (first build). But allow for higher versions, to support Python3.11 (not available in 1.21.4 yet)

opencv-contrib-python==4.5.5.62 # Last successful RPi build, also covers M1 with above pinned numpy (otherwise 4.6.0.62 would be required, but that has a bug with charuco boards). Python version not important, abi3 wheels

depthai-sdk==1.9.4

--extra-index-url https://artifacts.luxonis.com/artifactory/luxonis-depthai-data-local/wheels/

pyqt5>5,<5.15.6 ; platform_machine != "armv6l" and platform_machine != "armv7l" and platform_machine != "aarch64" and platform_machine != "arm64"

--extra-index-url https://artifacts.luxonis.com/artifactory/luxonis-python-snapshot-local/

depthai==2.21.2.0

Qt.py

scipy

jakaskerl

Hi @mdupre27
The depthai repo (except the calibration part) is a bit outdated. We are working on updating the requirements.
Go inside depthai-python repo and install examples/install_requirements.py this should give you the latest depthai version. Then go back to depthai repo and run the calibration.py.

Thanks,
Jaka

mdupre27

Thank you

mdupre27

So I alsohad to copy downloader and find_version.py

install_requirements.py isnow almost successfull, notifying me that some folders are not a git repository (??). not sure why as I just cloned it:

(luxonisenv) C:\Users\mdupr\Documents\depthai-luxonis\depthai>python install_requirements2.py

pip 23.3.2 from C:\Users\mdupr\AppData\Roaming\Python\Python311\site-packages\pip (python 3.11)

Requirement already satisfied: pip in c:\users\mdupr\appdata\roaming\python\python311\site-packages (23.3.2)

Requirement already satisfied: pyyaml in c:\users\mdupr\appdata\roaming\python\python311\site-packages (6.0.1)

Requirement already satisfied: requests in c:\users\mdupr\appdata\roaming\python\python311\site-packages (2.31.0)

Requirement already satisfied: numpy in c:\users\mdupr\appdata\roaming\python\python311\site-packages (1.26.3)

Requirement already satisfied: opencv-python in c:\users\mdupr\appdata\roaming\python\python311\site-packages (4.9.0.80)

Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\mdupr\appdata\roaming\python\python311\site-packages (from requests) (3.3.2)

Requirement already satisfied: idna<4,>=2.5 in c:\users\mdupr\appdata\roaming\python\python311\site-packages (from requests) (3.6)

Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\mdupr\appdata\roaming\python\python311\site-packages (from requests) (2.2.0)

Requirement already satisfied: certifi>=2017.4.17 in c:\users\mdupr\appdata\roaming\python\python311\site-packages (from requests) (2024.2.2)

fatal: not a git repository (or any of the parent directories): .git

Collecting depthai

Using cached depthai-2.24.0.0-cp311-cp311-win_amd64.whl.metadata (8.9 kB)

Using cached depthai-2.24.0.0-cp311-cp311-win_amd64.whl (10.4 MB)

Installing collected packages: depthai

Attempting uninstall: depthai

Found existing installation: depthai 2.24.0.0

Uninstalling depthai-2.24.0.0:

Successfully uninstalled depthai-2.24.0.0

Successfully installed depthai-2.24.0.0

if I try to run the demo, i get the following:

>python depthai_demo.py

Third party libraries failed to import: No module named 'xmltodict'

Run "python3 install_requirements.py" to install dependencies

In addition there is an issue with the requirements: If I try to run calibrate.py, it tells me that scipy is not installed. even if I install it manually with pip, I still get the error:

C:\Users\mdupr\Documents\depthai-luxonis\depthai>python calibrate.py -s 21.28 --board OAK-D-PRO -nx 41 -ny 23

Traceback (most recent call last):

File "C:\Users\mdupr\Documents\depthai-luxonis\depthai\calibrate.py", line 1181, in <module>

Main().run()

^^^^^^

File "C:\Users\mdupr\Documents\depthai-luxonis\depthai\calibrate.py", line 356, in __init__

self.aruco_dictionary = cv2.aruco.Dictionary_get(

^^^^^^^^^^^^^^^^^^^^^^^^

AttributeError: module 'cv2.aruco' has no attribute 'Dictionary_get'. Did you mean: 'Dictionary'?

I tried to uninstall both opencv-contrib-python and opencv-python and reinstall only opencv-contrib-python but neither helped.

jakaskerl

Hi @mdupre27
When pulling the repo make sure to use --recurse-submodules to update the submodules as well.
Also opencv-contrib-python==4.5.5.62 should work. I'll notify the team about the issue.

Thanks,
Jaka

mdupre27

Hi @jakaskerl

Yes, I literally followed those instructions, at the exception that I copy pasted some scriptis and folders from deht depthai-python to the local depthai repo.

git clone luxonis/depthai.git
cd depthai
git submodule update --init --recursive
python3 install_requirements.py

opencv-contrib-python==4.5.5.62

did not help

jakaskerl

Hi mdupre27
I went and tested this on a random WIN10 machine.

cloned depthai repo
cd into dir
git submodule update --init --recursive
python install_requirements.py
At this point, the calibration worked as expected.
Cloned the depthai-python repo and installed requirements and submodules
Calibration doesn't work since opencv=4.9smthg is installed.
Only by force reinstalling the opencv-contrib-python==4.5.5.62 and removing the opencv-python (standard package, also 4.90 something) was I able to get it working.

Are you still experiencing the sensorName issue? Could you paste the calibration dump?

Thanks and sorry for the inconvenience.

Jaka

mdupre27

Yes I started again fromn scratch. deleted depthai repo and conda environments:

conda create luxonisenv python=3.11
conda activate luxonisenv
git clone https://github.com/luxonis/depthai.git
cd depthai
git submodule update --init --recursive
python install_requirements.py

So far so good, only a few warnings saying some scripts are not on PATH.

(luxonisenv) C:\Users\mdupr\Documents\depthai-luxonis\depthai>python calibrate.py -s 2.26 --board OAK-D-PRO -nx 41 -ny 23

Saving dataset to: dataset
Traceback (most recent call last):
File "C:\Users\mdupr\Documents\depthai-luxonis\depthai\calibrate.py", line 1181, in <module>
Main().run()
File "C:\Users\mdupr\Documents\depthai-luxonis\depthai\calibrate.py", line 1170, in run
self.startPipeline()
File "C:\Users\mdupr\Documents\depthai-luxonis\depthai\calibrate.py", line 465, in startPipeline
pipeline = self.create_pipeline()
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\mdupr\Documents\depthai-luxonis\depthai\calibrate.py", line 554, in create_pipeline
sensorName = cam_info['sensorName']
~~~~~~~~^^^^^^^^^^^^^^
KeyError: 'sensorName'

I also have an issue with the demo, it is not just an issue with the calibration:

(luxonisenv) C:\Users\mdupr\Documents\depthai-luxonis\depthai>python depthai_demo.py

Using depthai module from: C:\Users\mdupr\AppData\Roaming\Python\Python311\site-packages\depthai.cp311-win_amd64.pyd

Depthai version installed: 2.21.2.0

C:\Users\mdupr\Documents\depthai-luxonis\depthai\depthai_demo.py:588: DeprecationWarning: sipPyTypeDict() is deprecated, the extension module should use sipPyTypeDictRef() instead

class WorkerSignals(QObject):

Setting up demo...

Available devices:

[0] 1944301071119B2E00 [X_LINK_UNBOOTED]

USB Connection speed: UsbSpeed.SUPER

Disabling depth...

Disabling depth preview...

Disabling depthRaw preview...

Disabling left preview...

Disabling rectifiedLeft preview...

Disabling right preview...

Disabling rectifiedRight preview...

IR drivers found on OAK-D Pro: ['LM3644 on bus 2']

--> Go to the `Depth` tab to enable!

[1944301071119B2E00] [2.4] [1.434] [ColorCamera(0)] [error] Camera not detected on socket: 0

I end up with this loading screen:

I checked i was usb usb3 cable, and tried different usb port on the computer…

mdupre27

Here is the result of the calibration_reader sccript:

(luxonisenv) C:\Users\mdupr\Documents\depthai-luxonis\depthai>python calibration_reader.py

RGB Camera Default intrinsics...

[[3116.619873046875, 0.0, 1918.7083740234375], [0.0, 3117.794189453125, 1094.328857421875], [0.0, 0.0, 1.0]]

3840

2160

RGB Camera Default intrinsics...

[[3116.619873046875, 0.0, 1918.7083740234375], [0.0, 3117.794189453125, 1094.328857421875], [0.0, 0.0, 1.0]]

3840

2160

RGB Camera resized intrinsics... 3840 x 2160

[[3.11661987e+03 0.00000000e+00 1.91870837e+03]

[0.00000000e+00 3.11779419e+03 1.09432886e+03]

[0.00000000e+00 0.00000000e+00 1.00000000e+00]]

RGB Camera resized intrinsics... 4056 x 3040

[[3.29192969e+03 0.00000000e+00 2.02663562e+03]

[0.00000000e+00 3.29316992e+03 1.53513489e+03]

[0.00000000e+00 0.00000000e+00 1.00000000e+00]]

LEFT Camera Default intrinsics...

[[802.652587890625, 0.0, 661.720703125], [0.0, 802.8941650390625, 410.12835693359375], [0.0, 0.0, 1.0]]

1280

800

LEFT Camera resized intrinsics... 1280 x 720

[[802.65258789 0. 661.72070312]

[ 0. 802.89416504 370.12835693]

[ 0. 0. 1. ]]

RIGHT Camera resized intrinsics... 1280 x 720

[[805.89227295 0. 650.75537109]

[ 0. 806.60412598 363.04296875]

[ 0. 0. 1. ]]

LEFT Distortion Coefficients...

k1: 14.230230331420898

k2: -15.112517356872559

p1: -0.0001133228579419665

p2: 0.0008751495624892414

k3: 2.171414613723755

k4: 13.8093900680542

k5: -14.002786636352539

k6: 1.3974621295928955

s1: 0.0

s2: 0.0

s3: 0.0

s4: 0.0

τx: 0.0

τy: 0.0

RIGHT Distortion Coefficients...

k1: 19.56604766845703

k2: -30.052560806274414

p1: -0.0003369998012203723

p2: 0.0008202568860724568

k3: 13.26058292388916

k4: 19.039215087890625

k5: -28.521833419799805

k6: 12.116994857788086

s1: 0.0

s2: 0.0

s3: 0.0

s4: 0.0

τx: 0.0

τy: 0.0

RGB FOV 68.7938003540039, Mono FOV 71.86000061035156

LEFT Camera stereo rectification matrix...

[[ 1.01241203e+00 -3.46723052e-03 -2.63160277e+01]

[ 7.61519624e-03 1.00429944e+00 -1.31946268e+01]

[ 1.29608461e-05 -8.72804373e-07 9.91692230e-01]]

RIGHT Camera stereo rectification matrix...

[[ 1.00834214e+00 -3.45128306e-03 -1.25964445e+01]

[ 7.58458317e-03 9.99680181e-01 -4.29836194e+00]

[ 1.29087435e-05 -8.68789926e-07 9.91860614e-01]]

Transformation matrix of where left Camera is W.R.T right Camera's optical center

[[ 9.99721467e-01 5.13254025e-04 -2.35957950e-02 -7.51707506e+00]

[-4.80024784e-04 9.99998868e-01 1.41391100e-03 2.54338868e-02]

[ 2.35964935e-02 -1.40219054e-03 9.99720573e-01 -9.92123187e-02]

[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 1.00000000e+00]]

Transformation matrix of where left Camera is W.R.T RGB Camera's optical center

[[ 9.99960601e-01 1.06213288e-03 -8.82022735e-03 -3.76098013e+00]

[-9.37739271e-04 9.99900222e-01 1.40954014e-02 6.92155957e-03]

[ 8.83431919e-03 -1.40865743e-02 9.99861717e-01 -2.14272916e-01]

[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 1.00000000e+00]]