Hi from Pollen Robotics 🙂 We just received our ToF module. Using this example we get a pretty nice looking depthmap (Except for the looping colors, but I assume that over a few meters the sensor is not accurate anymore) (We are using an OAK FFC-4P btw) However, when I try to mesure distances, they are pretty far off most of the time. For example Real distance (black circle) ~= 380 Real distance ~= 340 Real distance ~= 195 Real distance ~= 880 A few questions : I don't understand what the different parameters are doing, and the documentation is not very helpful to me (this is the first time I work with a ToF sensor). Could you explain what freqModUsed, avgPhaseShuffle, and minimumAmplitude are doing ? Am I right to expect millimeters out of the depth map ? Is there any calibration needed ? Could that have been degraded during transport ? Here is my code for depth mesurement in the image (at the position of the cursor), based on the provided example #!/usr/bin/env python3 import cv2 import depthai as dai import numpy as np pipeline = dai.Pipeline() cam_a = pipeline.create(dai.node.Camera) cam_a.setBoardSocket(dai.CameraBoardSocket.CAM_A) tof = pipeline.create(dai.node.ToF) # Configure the ToF node tofConfig = tof.initialConfig.get() tofConfig.depthParams.freqModUsed = dai.RawToFConfig.DepthParams.TypeFMod.MAX tofConfig.depthParams.avgPhaseShuffle = False tofConfig.depthParams.minimumAmplitude = 3.0 tof.initialConfig.set(tofConfig) cam_a.raw.link(tof.input) xout = pipeline.create(dai.node.XLinkOut) xout.setStreamName("depth") tof.depth.link(xout.input) last_mouse_pos = (0, 0) last_depth_mesurement = 0 def mouse_callback(event, x, y, flags, params): global last_mouse_pos last_mouse_pos = (x, y) cv2.namedWindow("colorized depth") cv2.setMouseCallback("colorized depth", mouse_callback) time_averaged_depth = [] # Connect to device and start pipeline with dai.Device(pipeline) as device: print('Connected cameras:', device.getConnectedCameraFeatures()) q = device.getOutputQueue(name="depth") while True: imgFrame = q.get() depth_map = imgFrame.getFrame() depth_downscaled = depth_map[::4] non_zero_depth = depth_downscaled[depth_downscaled != 0] # Remove invalid depth values if len(non_zero_depth) == 0: min_depth, max_depth = 0, 0 else: min_depth = np.percentile(non_zero_depth, 3) max_depth = np.percentile(non_zero_depth, 97) last_depth_mesurement = depth_map[last_mouse_pos[1]-5:last_mouse_pos[1]+5, last_mouse_pos[0]-5:last_mouse_pos[0]+5] last_depth_mesurement = np.mean(last_depth_mesurement) time_averaged_depth.append(last_depth_mesurement) time_averaged_depth = time_averaged_depth[20:] if len(time_averaged_depth) > 20 else time_averaged_depth print("mean :", np.mean(time_averaged_depth), "(std :", np.std(time_averaged_depth), ")") depth_colorized = np.interp(depth_map, (min_depth, max_depth), (0, 255)).astype(np.uint8) depth_colorized = cv2.applyColorMap(depth_colorized, cv2.COLORMAP_JET) depth_colorized = cv2.circle(depth_colorized, last_mouse_pos, 5, (0, 0, 0), -1) cv2.imshow("colorized depth", depth_colorized) if cv2.waitKey(1) == ord('q'): break if(window.hljsLoader && !document.currentScript.parentNode.hasAttribute('data-s9e-livepreview-onupdate')) { window.hljsLoader.highlightBlocks(document.currentScript.parentNode); } Thanks ! Antoine

Hi apirrone , We apologize for the inconvenience, but the full TOF decoding is not yet implemented, and we are working on it at the moment. It should be ready in firmware in a few week, so you'll be able to just update the library (pip install depthai -U) and have accurate depth. Kind regards, Erik

Thanks erik for the reply, please keep me updated when the update is ready! Just curious, how is it that it "kind of works" right now, but with bad accuracy? The colorized depth map looks really good and continuous 😅

Hi apirrone , That's the thing - we get some data out, but full decoding and error correction aren't implemented. The depth does look decent (at least the corners are very sharp), but if you'd visualize the pointcloud it's completely off.

Hi @erik , Any news on the firmware ? Thanks ! Antoine

Hi @apirrone , We got the decoding mostly working, there's just some offset left that we haven't figured out. Below is an A4 paper sheet and its size - about 10% dimension mismatch. We believe we'll have it ready this month. Thanks, Erik

Hi @erik , any news ? We are very eager to try it out ! 🙂 Thanks, Antoine

Hi apirrone We need to finalize a few things, I'd say roughly about 2 weeks.. Thanks, Jaka

jakaskerl hey any update on the ToF sensor finalization so its accurate and ready to be used in depthai?

ToF bad accuracy

apirrone

jakaskerl This script has never worked for me, that's why I made a custom calibration process.

I tried again still, in the tof_calib branch, with a fresh virtual environment and I have to iteratively comment out / edit / add things in the script to move on with the process. For example, for some reason, these parameters don't exist in this branch


                    tof_config.enablePhaseShuffleTemporalFilter = True
                    tof_config.enableFPPNCorrection = True
                    tof_config.enableOpticalCorrection = True
                    tof_config.enableWiggleCorrection = True
                    tof_config.enableTemperatureCorrection = True
                    tof_config.phaseUnwrappingLevel = 4
                    tof_config.phaseUnwrapErrorThreshold = 1000

while they exist here 2.28.0.0.dev0+76294530fd0bf91b09469490a1d14871c374630a.

Same for intensity, which means I'm stuck here luxonis/depthaiblob/28f2bd654923e24019f00248cf03925866b107af/calibrate.py#L608

So I recorded the images and computed the calibration using my usual process and I still have alignement issues.

This is where I flash the calibration : pollen-robotics/pollen-visionblob/01f1901a95f44e052b2953e4453247c6d7aa6b47/pollen_vision/pollen_vision/camera_wrappers/depthai/wrapper.py#L260

Now I get the intrinsics and extrinsics from my calibration process
It seems better than before :

But there is still a misalign when my arm is closer for example

Now, when I undistort the rgb first

jakaskerl Undistort the RGB image first. More on this here.

If doesn't change much because most of the distortion is on the edge of the FOV and the ToF's FOV is is much smaller than our fisheye lenses.

Also, the ToF can get pretty noisy sometimes (as you can see on top of the image below):

Do you have any advice ?

Thanks

jakaskerl

apirrone
Maybe the custom calib process you are using produces errors, so I am going to try to fix the tof_calib process...

apirrone I tried again still, in the tof_calib branch, with a fresh virtual environment and I have to iteratively comment out / edit / add things in the script to move on with the process. For example, for some reason, these parameters don't exist in this branch

checkout this branch, but keep depthai==2.28, set numpy==1.26.4 and opencv-contrib-python==4.6.0.66.

Then run the calibration and try again. You should not have to remove anything.

Thanks,
Jaka

apirrone

@jakaskerl I have a new problem

I was messing around with calibration, and suddenly (well I guess It happened after I flashed a new test calibration), the raw intensity output of the ToF is all distorted :

This obviously looks like bad distortion coefficients, but now I can't get a raw intensity image without this distortion ... Even with this minimal code :

import depthai as dai
import cv2

pipeline = dai.Pipeline()

cam_tof = pipeline.create(dai.node.Camera)
cam_tof.setFps(30)

cam_tof.setBoardSocket(dai.CameraBoardSocket.CAM_D)

tof = pipeline.create(dai.node.ToF)
cam_tof.raw.link(tof.input)

output = pipeline.create(dai.node.XLinkOut)
output.setStreamName("intensity")

tof.intensity.link(output.input)

device = dai.Device(pipeline)

q = device.getOutputQueue(name="intensity", maxSize=8, blocking=False)

while True:
    data = q.get()
    cv2.imshow("intensity", data.getCvFrame())
    cv2.waitKey(1)

I was also messing around with depthai versions, I built the latest version of develop from source :

2.28.0.0.dev0+d3b92a92782894b33015d97d435451ff35532d06

It's more recent than the version you told me to use I guess

Luxonis-Alex depthai version currently

So I went back to this version 76294530fd0bf91b09469490a1d14871c374630a and same thing.

So now I can't get raw data for calibration. Any idea what happened ?

apirrone

Ok I saw this in the latest changelog

Adding this solved my problem

« Previous Page