Difference in depth measurement for multiple cameras

Tsjarly

Hello,

We have 3 Oak-D pro cameras and while working on a camera-to-camera calibration script I noticed a strange offset in depth measurement. So we made a setup where the camera position is fixed and I move a specific object to different distances. I did the same experiment for 3 Oak-D pro cameras on the exact same position, measuring the same points.
for a distance of +- 40 cm, there is a maximum difference of 13mm between cameras, for 60cm this is 22mm, for 80cm this is 32mm and for 100cm this is even 53mm (always the same camera having the largest offset)

Is it possible that the different cameras of the same model have a different depth measurement "offset" or something? or is this something that should be calibrated as well. If so, how would I go about that?

jakaskerl

Hi Tsjarly
I'd say it is likely just a bad calibration. Perhaps during transport, the cameras shifted or the outside environment caused the device to bend slightly which would result in poorer depth perception. I think you can easily fix this by recalibrating the device.

Thanks,
Jaka

Tsjarly

Hey @jakaskerl,

When I try to run the calibrate.py script, it opens up the initial prompt, but after pressing the spacebar once, it gives me this error:

python3 calibrate.py -s 2.3 -brd bw1098obc -db -ih

[1844301031A1820E00] [1.2.2.4.3] [0.822] [ColorCamera(4)] [warning] Unsupported resolution set for detected camera OV9782, needs 800_P or 720_P. Defaulting to 800_P

Starting image capture. Press the [ESC] key to abort.

Will take 13 total images, 1 per each polygon.

800 1280

Traceback (most recent call last):

File "calibrate.py", line 674, in <module>

Main().run()

File "calibrate.py", line 666, in run

self.capture_images()

File "calibrate.py", line 393, in capture_images

frame = packet[1].getCvFrame()

cv2.error: OpenCV(4.5.5) /io/opencv/modules/imgproc/src/color.simd_helpers.hpp:108: error: (-215:Assertion failed) sz.width % 2 == 0 && sz.height % 3 == 0 in function 'CvtHelper'

When I disable rgb calilbration, the code works.

After running this script, will this calibration be stored on the device itself, or should I copy some files into my project folder / should I run this from my project folder?

jakaskerl

Hi Tsjarly
The calibration data is stored on the device itself. However, you can also back up the calibration data to a file. The calibration script you're using should create a backup file named depthai_calib_backup.json in the same directory. You can use this file to restore the calibration data to the device if needed.

The error you're encountering seems to be related to the OpenCV installation on your host machine. You can check this by running the following command:

python3 -c "import cv2; import numpy as np; blank_image = np.zeros((500,500,3), np.uint8); cv2.imshow('s', blank_image); cv2.waitKey(0)"

If a window is not displayed, or if you get an "Illegal instruction" result, this means there is a problem with the OpenCV install. You can try reinstalling OpenCV with the following command:

python3 -m pip install opencv-python --force-reinstall

If the problem persists, you might want to check the camera streams using the cam_test.py script. This script allows you to quickly test whether cameras are working as expected. You can specify which camera types to use with the --cameras argument.

Thanks,
Jaka

Tsjarly

Hi @jakaskerl

I indeed found that they are stored on the device itself, and is it correct that these jsons could be named like 'mxid'.json? It at least seems like the correct files.

The test command prompts a small black image, so that seems alright.
(I tried reinstalling opencv anyways, but without luck, it even gave me a new error:

Traceback (most recent call last):

File "calibrate.py", line 672, in <module>

Main().run()

File "calibrate.py", line 131, in __init__

self.aruco_dictionary = cv2.aruco.Dictionary_get(

AttributeError: module 'cv2.aruco' has no attribute 'Dictionary_get'

I tried to run the cam_test.py but also this came with an error..

Traceback (most recent call last):

File "cam_test.py", line 114, in <module>

import cam_test_gui

File "/home/jarl/Desktop/Luxonis/depthai-python/utilities/cam_test_gui.py", line 2, in <module>

from PyQt5 import QtCore, QtGui, QtWidgets

ModuleNotFoundError: No module named 'PyQt5.sip'

Tsjarly

hi @jakaskerl

I managed to get the cam_test script running, but when selecting rgb camera as color, I get the following error when connecting:

Enabled cameras:

rgb : color

left : mono

right : mono

camd : color

DepthAI version: 2.20.2.0

DepthAI path: /home/jarl/Luxonis/venv/lib/python3.8/site-packages/depthai.cpython-38-x86_64-linux-gnu.so

Traceback (most recent call last):

File "cam_test.py", line 162, in <module>

'1012': dai.ColorCameraProperties.SensorResolution.THE_1352X1012,

AttributeError: type object 'depthai.SensorResolution' has no attribute 'THE_1352X1012'

Tsjarly

This is with the current settings:

Same error when running without GUI

jakaskerl

Hi Tsjarly
I'd try updating the depthai-python repo and then running python3 install_requirements.py. You are currently running depthai 2.20.2.0 which is outdated and might not have all the features you require.

Thanks,
Jaka

Tsjarly

Hi @jakaskerl

I did a full reinstall of everything, new clone of repo, new venv, new base python interpreter, did install_requirements.

When running the cam_test it produces the following output:

QObject::moveToThread: Current thread (0x556f262bfb10) is not the object's thread (0x556f26468550).

Cannot move to target thread (0x556f262bfb10)

qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "/home/jarl/Desktop/Luxonis Fresh/venv/lib/python3.8/site-packages/cv2/qt/plugins" even though it was found.

This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: xcb, eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, wayland-egl, wayland, wayland-xcomposite-egl, wayland-xcomposite-glx, webgl.

Stack trace (most recent call last):

#28 Object "[0xffffffffffffffff]", at 0xffffffffffffffff, in

#27 Object "python3.8", at 0x556f24a6fc24, in _start

#26 Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f6f88629e3f, in __libc_start_main

#25 Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f6f88629d8f, in

#24 Object "python3.8", at 0x556f24a6fd2c, in Py_BytesMain

#23 Object "python3.8", at 0x556f24a7037f, in Py_RunMain

#22 Object "python3.8", at 0x556f2496da79, in PyRun_SimpleFileExFlags

#21 Object "python3.8", at 0x556f2496dca7, in PyRun_FileExFlags

#20 Object "python3.8", at 0x556f24aa4d82, in

#19 Object "python3.8", at 0x556f24aa4de8, in

#18 Object "python3.8", at 0x556f24a948a6, in PyEval_EvalCode

#17 Object "python3.8", at 0x556f24a1fac2, in _PyEval_EvalCodeWithName

#16 Object "python3.8", at 0x556f24a25368, in _PyEval_EvalFrameDefault

#15 Object "python3.8", at 0x556f249ca269, in _PyFunction_Vectorcall

#14 Object "python3.8", at 0x556f24a257fe, in _PyEval_EvalFrameDefault

#13 Object "python3.8", at 0x556f249c9997, in _PyObject_MakeTpCall

#12 Object "python3.8", at 0x556f249fc572, in

#11 Object "/home/jarl/Desktop/Luxonis Fresh/venv/lib/python3.8/site-packages/PyQt5/sip.cpython-38-x86_64-linux-gnu.so", at 0x7f6f7b7f3861, in

#10 Object "/home/jarl/Desktop/Luxonis Fresh/venv/lib/python3.8/site-packages/PyQt5/QtWidgets.abi3.so", at 0x7f6f3648bffa, in

#9 Object "/home/jarl/Desktop/Luxonis Fresh/venv/lib/python3.8/site-packages/PyQt5/QtWidgets.abi3.so", at 0x7f6f3648bf1c, in

#8 Object "/home/jarl/Desktop/Luxonis Fresh/venv/lib/python3.8/site-packages/PyQt5/Qt5/lib/libQt5Widgets.so.5", at 0x7f6f35967aa8, in QApplicationPrivate::init()

#7 Object "/home/jarl/Desktop/Luxonis Fresh/venv/lib/python3.8/site-packages/PyQt5/Qt5/lib/libQt5Gui.so.5", at 0x7f6f3c76accb, in QGuiApplicationPrivate::init()

#6 Object "/home/jarl/Desktop/Luxonis Fresh/venv/lib/python3.8/site-packages/PyQt5/Qt5/lib/libQt5Core.so.5", at 0x7f6f3faa4fe2, in QCoreApplicationPrivate::init()

#5 Object "/home/jarl/Desktop/Luxonis Fresh/venv/lib/python3.8/site-packages/PyQt5/Qt5/lib/libQt5Gui.so.5", at 0x7f6f3c7682ac, in QGuiApplicationPrivate::createEventDispatcher()

#4 Object "/home/jarl/Desktop/Luxonis Fresh/venv/lib/python3.8/site-packages/PyQt5/Qt5/lib/libQt5Gui.so.5", at 0x7f6f3c767f9f, in QGuiApplicationPrivate::createPlatformIntegration()

#3 Object "/home/jarl/Desktop/Luxonis Fresh/venv/lib/python3.8/site-packages/PyQt5/Qt5/lib/libQt5Core.so.5", at 0x7f6f3f8929bb, in QMessageLogger::fatal(char const*, ...) const

#2 Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f6f886287f2, in abort

#1 Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f6f88642475, in raise

#0 Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f6f886969fc, in pthread_kill

Aborted (Signal sent by tkill() 46835 1000)

Aborted (core dumped)

I tried to install opencv-python-headless, which then worked, but produced the following error:

Enabled cameras:

rgb : color

left : mono

right : mono

camd : color

DepthAI version: 2.23.0.0

DepthAI path: /home/jarl/Desktop/Luxonis Fresh/venv/lib/python3.8/site-packages/depthai.cpython-38-x86_64-linux-gnu.so

cam_test.py:298: DeprecationWarning: Use constructor taking 'UsbSpeed' instead

with dai.Device(*dai_device_args) as device:

[1844301031A1820E00] [1.2.2] [0.699] [ColorCamera(9)] [error] Camera not detected on socket: 3

[1844301031A1820E00] [1.2.2] [0.700] [ColorCamera(3)] [warning] Unsupported resolution set for detected camera OV9782, needs 800_P or 720_P. Defaulting to 800_P

Connected cameras:

-socket CAM_A : OV9782 1280 x 800 focus:fixed - COLOR MONO

-socket CAM_B : OV9282 1280 x 800 focus:fixed - MONO

-socket CAM_C : OV9282 1280 x 800 focus:fixed - MONO

USB speed: SUPER

IR drivers: [('LM3644', 2, 99)]

Cam: rgb left right camd [host | capture timestamp]

FPS: 0.00| 0.00 0.00| 0.00 0.00| 0.00 0.00| 0.00 Traceback (most recent call last):

File "cam_test.py", line 460, in <module>

key = cv2.waitKey(1)

cv2.error: OpenCV(4.8.1) /io/opencv/modules/highgui/src/window.cpp:1338: error: (-2:Unspecified error) The function is not implemented. Rebuild the library with Windows, GTK+ 2.x or Cocoa support. If you are on Ubuntu or Debian, install libgtk2.0-dev and pkg-config, then re-run cmake or configure script in function 'cvWaitKey'

(libgtk2.0-dev and pkg-config are both installed and up to date)

Also when installing the requirements, it mentions that
Traceback (most recent call last):

File "<string>", line 1, in <module>

File "/home/jarl/Python-3.8.0/Lib/tkinter/__init__.py", line 36, in <module>

import _tkinter # If this fails your Python may not be configured for Tk

ModuleNotFoundError: No module named '_tkinter'

Missing `python3-tk` on the system. Install with `sudo apt install python3-tk`

but also this is installed and up to date, however it keeps mentioning this..
I also tried updating the opencv-contrib-python to latest package, but same error as the first

Tsjarly

Hi @jakaskerl

I also did a complete reinstall of the general Lunoxis depthai (with the calibrate.py), running this command:

python3 calibrate.py -s 2.3 -db -brd bw1098obc -ih

as suggested in the documentations will raise the error that

calibrate.py: error: argument -db/--defaultBoard: expected one argument

when I leave the -db out of the command, it will run and start the calibration prompt. However, it mentions that I have to take 39 images in total now (instead of 13 earlier), and the footage shows some green detections of the markers (earlier there was a specific field where I had to hold the marker, which is gone now). Is this the way it should be working?

Tsjarly

Okay I managed to get it working, it was just a bunch of settings that I had to go over.
One more question for now, I was wondering whether the script only works for the predefined charuco boards in the repository, or that it would work for any (different size, different amount of squares, different dicts (5x5)) board.
We have a charuco board of 9x12 squares with dict 5x5, checker size 30mm, marker size 23mm. I dont know what the size would be though, are these diagonally measured?

jakaskerl

Tsjarly
You can use different charuco boards, just make sure to specify the -nx and -ny args to match the board. You can use calibrate.py --help to find out more.

Thanks,
Jaka