Oak-d-sr-poe calibration

gdeanrexroth
Update- I came across this link:https://luxonis-depthai-hardware.readthedocs-hosted.com/en/latest/pages/guides/calibration/#tof-calibration
I followed the git command line"git checkout new_tof_calib", the terminal then outputed this "warning: unable to rmdir 'depthai_calibration': Directory not empty

Updating files: 100% (316/316), done.

M resources/depthai_boards

branch 'new_tof_calib' set up to track 'origin/new_tof_calib'."

i ran the calibrate.py file and got a new output within terminal. Here is the result:
"Starting image processing

Starting image processingxxccx

11

squareX is 11

<------------Calibrating left ------------>

Device closed in exception..

ERROR: Images not read correctly, check directory

Traceback (most recent call last):

File "S:\DEPT\SVM4\Shared\Crossfunctional_Work\Projects\DepthCameras\LuxonisDepthAI\test_run\depthai\calibrate.py", line 968, in calibrate

status, result_config = stereo_calib.calibrate(

                        ^^^^^^^^^^^^^^^^^^^^^^^

File "S:\DEPT\SVM4\Shared\Crossfunctional_Work\Projects\DepthCameras\LuxonisDepthAI\test_run\depthai\depthai_helpers\calibration_utils.py", line 121, in calibrate

ret, intrinsics, dist_coeff, _, _, size = self.calibrate_intrinsics(

                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^

File "S:\DEPT\SVM4\Shared\Crossfunctional_Work\Projects\DepthCameras\LuxonisDepthAI\test_run\depthai\depthai_helpers\calibration_utils.py", line 258, in calibrate_intrinsics

assert len(

       ^^^^

AssertionError: ERROR: Images not read correctly, check directory"
Beside switching to the new branch there is nothing that has changed within the code.

Brandon
After running the commands that you mentioned, The traceback changed:

python calibrate.py -db -nx 13 -ny 7 -c 1 -cd 0 -s 4 -ms 3 -brd OAK-D-SR-POE(command i entered in terminal)

[{socket: CAM_A, sensorName: S5K33D, width: 640, height: 480, orientation: ROTATE_180_DEG, supportedTypes: [TOF], hasAutofocus: 0, hasAutofocusIC: 0, name: tof}, {socket: CAM_B, sensorName: OV9782, width: 1280, height: 800, orientation: ROTATE_180_DEG, supportedTypes: [COLOR], hasAutofocus: 0, hasAutofocusIC: 0, name: left}, {socket: CAM_C, sensorName: OV9782, width: 1280, height: 800, orientation: ROTATE_180_DEG, supportedTypes: [COLOR], hasAutofocus: 0, hasAutofocusIC: 0, name: right}]

Cam: rgb and focus: False

Cam: left and focus: False

Cam: right and focus: False

Camera type ToF

Traceback (most recent call last):

File "S:\DEPT\SVM4\Shared\Crossfunctional_Work\Projects\DepthCameras\LuxonisDepthAI\test_run\depthai\calibrate.py", line 1207, in <module>

File "S:\DEPT\SVM4\Shared\Crossfunctional_Work\Projects\DepthCameras\LuxonisDepthAI\test_run\depthai\calibrate.py", line 333, in init

**pipeline = self.create_pipeline()**

           **^^^^^^^^^^^^^^^^^^^^^^**

File "S:\DEPT\SVM4\Shared\Crossfunctional_Work\Projects\DepthCameras\LuxonisDepthAI\test_run\depthai\calibrate.py", line 445, in create_pipeline

**tof_config.depthParams.freqModUsed = dai.RawToFConfig.DepthParams.TypeFMod.MIN**

                                     **^^^^^^^^^^^^^^^^^^^^^^^^^^^^**

AttributeError: type object 'depthai.RawToFConfig' has no attribute 'DepthParams'

gdeanrexroth
However when I commented out line 445, and ran the same command line. The interface to capture the images pops up on the screen. But the results are different:
Cam: rgb and focus: False

Cam: left and focus: False

Cam: right and focus: False

Camera type ToF

Starting image capture. Press the [ESC] key to abort.

Will take 13 total images, 1 per each polygon.

left 0:01:25.766316

new minimum: {'ts': 0.0, 'indicies': {'left': 0}} min required: 0.2

synced frames: None

right 0:01:25.766330

new minimum: {'ts': 1.3999999993075107e-05, 'indicies': {'left': 0, 'right': 0}} min required: 0.2

synced frames: None

rgb 0:01:25.667397

new minimum: {'ts': 0.09893300000000238, 'indicies': {'left': 0, 'right': 0, 'rgb': 0}} min required: 0.2

synced frames: None

left 0:01:25.866318

new minimum: {'ts': 0.09893300000000238, 'indicies': {'left': 0, 'right': 0, 'rgb': 0}} min required: 0.2

synced frames: None

right 0:01:25.866340

rgb 0:01:25.767397

new minimum: {'ts': 0.09893300000000238, 'indicies': {'left': 0, 'right': 0, 'rgb': 0}} min required: 0.2

new minimum: {'ts': 0.0010949999999922966, 'indicies': {'left': 0, 'right': 0, 'rgb': 1}} min required: 0.2

synced frames: None

left 0:01:25.966316

new minimum: {'ts': 0.09893300000000238, 'indicies': {'left': 0, 'right': 0, 'rgb': 0}} min required: 0.2

new minimum: {'ts': 0.0010949999999922966, 'indicies': {'left': 0, 'right': 0, 'rgb': 1}} min required: 0.2

synced frames: None

right 0:01:25.966328

new minimum: {'ts': 0.09893300000000238, 'indicies': {'left': 0, 'right': 0, 'rgb': 0}} min required: 0.2

new minimum: {'ts': 0.0010949999999922966, 'indicies': {'left': 0, 'right': 0, 'rgb': 1}} min required: 0.2

synced frames: None

rgb 0:01:25.867395

new minimum: {'ts': 0.09893300000000238, 'indicies': {'left': 0, 'right': 0, 'rgb': 0}} min required: 0.2

new minimum: {'ts': 0.0010949999999922966, 'indicies': {'left': 0, 'right': 0, 'rgb': 1}} min required: 0.2

synced frames: None

rgb 0:01:25.967395

new minimum: {'ts': 0.09893300000000238, 'indicies': {'left': 0, 'right': 0, 'rgb': 0}} min required: 0.2

new minimum: {'ts': 0.0010949999999922966, 'indicies': {'left': 0, 'right': 0, 'rgb': 1}} min required: 0.2

synced frames: None

right 0:01:26.066337

left 0:01:26.066318

new minimum: {'ts': 0.09893300000000238, 'indicies': {'left': 0, 'right': 0, 'rgb': 0}} min required: 0.2

new minimum: {'ts': 0.0010949999999922966, 'indicies': {'left': 0, 'right': 0, 'rgb': 1}} min required: 0.2

synced frames: None

left 0:01:26.166316

new minimum: {'ts': 0.09893300000000238, 'indicies': {'left': 0, 'right': 0, 'rgb': 0}} min required: 0.2

new minimum: {'ts': 0.0010949999999922966, 'indicies': {'left': 0, 'right': 0, 'rgb': 1}} min required: 0.2

synced frames: None

right 0:01:26.166329

new minimum: {'ts': 0.09893300000000238, 'indicies': {'left': 0, 'right': 0, 'rgb': 0}} min required: 0.2

new minimum: {'ts': 0.0010949999999922966, 'indicies': {'left': 0, 'right': 0, 'rgb': 1}} min required: 0.2

synced frames: None

rgb 0:01:26.067393

new minimum: {'ts': 0.09893300000000238, 'indicies': {'left': 0, 'right': 0, 'rgb': 0}} min required: 0.2

new minimum: {'ts': 0.0010949999999922966, 'indicies': {'left': 0, 'right': 0, 'rgb': 1}} min required: 0.2

Returning synced messages with error: 0.0010949999999922966 {'left': 0, 'right': 0, 'rgb': 1}

synced frames: {'left': <depthai.ImgFrame object at 0x0000019B3839E830>, 'right': <depthai.ImgFrame object at 0x0000019B59F61530>, 'rgb': <depthai.ImgFrame object at 0x0000019B59F03BF0>}

Timestamp of left is 0:01:25.766316

Timestamp of right is 0:01:25.766330

Timestamp of rgb is 0:01:25.767397

Traceback (most recent call last):

File "S:\test_run\depthai\calibrate.py", line 1207, in <module>

**Main().run()**

File "S:\test_run\depthai\calibrate.py", line 1196, in run

**self.capture_images_sync()**

File "S:\DEPT\SVM4\Shared\Crossfunctional_Work\Projects\DepthCameras\LuxonisDepthAI\test_run\depthai\calibrate.py", line 590, in capture_images_sync

**gray_frame = cv2.cvtColor(frameMsg.getCvFrame(), cv2.COLOR_BGR2GRAY)**

             **^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^**

cv2.error: OpenCV(4.5.5) d:\a\opencv-python\opencv-python\opencv\modules\imgproc\src\color.simd_helpers.hpp:92: error: (-2:Unspecified error) in function '__cdecl cv::impl::`anonymous-namespace'::CvtHelper<struct cv::impl::`anonymous namespace'::Set<3,4,-1>,struct cv::impl::A0xa96199bf::Set<1,-1,-1>,struct cv::impl::A0xa96199bf::Set<0,2,5>,2>::CvtHelper(const class cv::InputArray &,const class cv::OutputArray &,int)'

> Invalid number of channels in input image:

> 'VScn::contains(scn)'

> where

> 'scn' is 1

gdeanrexroth
Update: Disregard the above comments. I followed the git submodule command that you mentioned. It created the "new_tof_calib" branch on my local computer. As i was going through the code and debugging it. Most of the methods and functions are compatible with different python libraries that are not installed on my computer. After going through various documentation, I changed my conda environment based by the updated script. Now that it is updated, I was able to run the code and display the interface as coded to do. But whenever I click the space bar to capture the images, it prompt this"python calibrate.py -db charuco_24inch_13x7 -nx 13 -ny 7 -c 1 -cd 0 -s 4 -ms 3 -brd OAK-D-SR-POE

Cam: rgb and focus: False

Cam: left and focus: False

Cam: right and focus: False

Saving dataset to: dataset

Sensor name for left is OV9782

Sensor name for right is OV9782

Camera type ToF

Starting image capture. Press the [ESC] key to abort.

Will take 13 total images.

Start capturing...

new minimum: {'ts': 0.09893999999999892, 'indicies': {'left': 0, 'right': 0, 'rgb': 0}} min required: 0.2

new minimum: {'ts': 0.0010839999999987526, 'indicies': {'left': 0, 'right': 0, 'rgb': 1}} min required: 0.2

Time stamp of left is 3:23:16.393060

Markers count ... 39

Total markers needed -> 18

py: Saved image as: dataset\left\p0_0.png

Status of left is True

Time stamp of right is 3:23:16.393072

Markers count ... 25

Total markers needed -> 18

py: Saved image as: dataset\right\p0_0.png

Status of right is True

Time stamp of rgb is 3:23:16.394132

Markers count ... 0

Total markers needed -> 18

Status of rgb is False

py: Capture failed, unable to find chessboard! Fix position and press spacebar again"

jakaskerl
Before you made your comment I have tried to update my submodule through the depthai_calibration. then i ran the command "python calibrate.py -db 24_charuco13x7 -nx 13 -ny 7 -c 1 -cd 0 -s 4 -ms 3 -brd OAK-D-SR-POE". But my results are below. I am on the tof_calib branch. The interface which displays the info popped up and it allowed me to click space. But the errors prompted at the bottom. Is there a loop within the code that could be causing this?

Cam: rgb and focus: False

Cam: left and focus: False

Cam: right and focus: False

Saving dataset to: dataset

Camera type ToF

Sensor name for right is OV9782

Camera type ToF

Starting image capture. Press the [ESC] key to abort.

Will take 13 total images.

[14442C10A1C6AECF00] [169.254.1.222] [7.072] [ToF(1)] [error] Unexpected input image size 1280 x 800, maybe connected camera is not ToF?

jakaskerl
Baseline for the camera is 20mm and the ToF is 20cm - 5m. How does this affect the AOC and FOV? Does the FOV and AOC affect point cloud and pcds distribution? If i could implement them within the code that captures the pcd, would that potentially help?

jakaskerl
1. I have tried to changing the value for it based off the distance between the camera and the object. I am seeing what gives me the best result.
2. Main.py method/usage of open3d is a good reference, but the script i have to visuzlaize the pcd is this
import open3d as o3d
# Path to the .pcd file
pcd_file_path = r'S:gcolor_pcd_1.pcd'# Load the point cloudpcd = o3d.io.read_point_cloud(pcd_file_path)
# Visualize the point cloud
o3d.visualization.draw_geometries([pcd], window_name="ToF Point Cloud")
3. The on device point cloud method from that link does not work. I believe you have sent me a link to some an updated version of the script and it worked. I have moidified the script some previous times to test but the error of this:
inMessage = q.get()
Always pop up in my terminal.

Could it possibly be the way I am utilizing open3d? I have tried methods for visualizatio , which stitches two point cloud into one. But I am still left left with unnecessary noise. Adjusting the depthscale and depth_trunc does somewhat changes the output of my pcd whenever i visualize it. The snippet is apart of my code. I am using open3d here to convert the depth from the tof sensor and color camera. Currently the distance from the object to the camera is roughly 170 cm apart. For testing purposes I specifically want to only capture that distance, but I am still getting some type of noise at the type view and unnecessary noise from the side views:
# Convert depth image to Open3D format

            **depth_o3d = o3d.geometry.Image(depth_map)**

            **color_o3d = o3d.geometry.Image(cv2.cvtColor(color_frame_resized, cv2.COLOR_BGR2RGB))**

            **# Generate and save colored point cloud**

            **rgbd_image = o3d.geometry.RGBDImage.create_from_color_and_depth(**

                **color_o3d, depth_o3d, depth_scale=1700.0, depth_trunc=3.7, convert_rgb_to_intensity=False**

            **)**

jakaskerl
The pcd that I captured using the tof sensor and color camera was tested in another way. I placed the captured pcd into a online website that converts it to a point cloud. I noticed that it displayed the same as my code.
I will look at the tof configuration again along with the point cloud code. I am somewhat close to figuring this out but I am still lost in some areas. Here is the screenshot of the website pcd.

The json below is the saved depth information of the captured pcd from my script.
"
{

**"class_name" : "PinholeCameraParameters",**

**"extrinsic" :** 

**[**

	**0.97137093405355701,**

	**0.080079400112124194,**

	**0.22366447673603115,**

	**0.0,**

	**-0.17295492005240018,**

	**0.88380774499819115,**

	**0.43470733316897259,**

	**0.0,**

	**-0.16286529435575947,**

	**-0.46094593995271788,**

	**0.87235528102689741,**

	**0.0,**

	**-0.019386541654994344,**

	**-0.24191574413614514,**

	**1.0362066902315488,**

	**1.0**

**],**

**"intrinsic" :** 

**{**

	**"height" : 1009,**

	**"intrinsic_matrix" :** 

	**[**

		**873.8196324184986,**

		**0.0,**

		**0.0,**

		**0.0,**

		**873.8196324184986,**

		**0.0,**

		**959.5,**

		**504.0,**

		**1.0**

	**],**

	**"width" : 1920**

**},**

**"version_major" : 1,**

**"version_minor" : 0**

}
"