Hi from Pollen Robotics 🙂

We just received our ToF module. Using this example we get a pretty nice looking depthmap

(Except for the looping colors, but I assume that over a few meters the sensor is not accurate anymore)

(We are using an OAK FFC-4P btw)

However, when I try to mesure distances, they are pretty far off most of the time. For example

Real distance (black circle) ~= 380

Real distance ~= 340

Real distance ~= 195

Real distance ~= 880

A few questions :

  • I don't understand what the different parameters are doing, and the documentation is not very helpful to me (this is the first time I work with a ToF sensor). Could you explain what freqModUsed, avgPhaseShuffle, and minimumAmplitude are doing ?
  • Am I right to expect millimeters out of the depth map ?
  • Is there any calibration needed ? Could that have been degraded during transport ?

Here is my code for depth mesurement in the image (at the position of the cursor), based on the provided example

#!/usr/bin/env python3

import cv2
import depthai as dai
import numpy as np

pipeline = dai.Pipeline()

cam_a = pipeline.create(dai.node.Camera)
cam_a.setBoardSocket(dai.CameraBoardSocket.CAM_A)

tof = pipeline.create(dai.node.ToF)

# Configure the ToF node
tofConfig = tof.initialConfig.get()
tofConfig.depthParams.freqModUsed = dai.RawToFConfig.DepthParams.TypeFMod.MAX
tofConfig.depthParams.avgPhaseShuffle = False
tofConfig.depthParams.minimumAmplitude = 3.0
tof.initialConfig.set(tofConfig)
cam_a.raw.link(tof.input)

xout = pipeline.create(dai.node.XLinkOut)
xout.setStreamName("depth")
tof.depth.link(xout.input)

last_mouse_pos = (0, 0)
last_depth_mesurement = 0

def mouse_callback(event, x, y, flags, params):
    global last_mouse_pos
    last_mouse_pos = (x, y)


cv2.namedWindow("colorized depth")
cv2.setMouseCallback("colorized depth", mouse_callback)
time_averaged_depth = []

# Connect to device and start pipeline
with dai.Device(pipeline) as device:
    print('Connected cameras:', device.getConnectedCameraFeatures())
    q = device.getOutputQueue(name="depth")

    while True:
        imgFrame = q.get()
        depth_map = imgFrame.getFrame()

        depth_downscaled = depth_map[::4]
        non_zero_depth = depth_downscaled[depth_downscaled != 0] # Remove invalid depth values
        if len(non_zero_depth) == 0:
            min_depth, max_depth = 0, 0
        else:
            min_depth = np.percentile(non_zero_depth, 3)
            max_depth = np.percentile(non_zero_depth, 97)

        last_depth_mesurement = depth_map[last_mouse_pos[1]-5:last_mouse_pos[1]+5, last_mouse_pos[0]-5:last_mouse_pos[0]+5]
        last_depth_mesurement = np.mean(last_depth_mesurement)
        time_averaged_depth.append(last_depth_mesurement)

        time_averaged_depth = time_averaged_depth[20:] if len(time_averaged_depth) > 20 else time_averaged_depth
        print("mean :", np.mean(time_averaged_depth), "(std :", np.std(time_averaged_depth), ")")

        depth_colorized = np.interp(depth_map, (min_depth, max_depth), (0, 255)).astype(np.uint8)
        depth_colorized = cv2.applyColorMap(depth_colorized, cv2.COLORMAP_JET)

        depth_colorized = cv2.circle(depth_colorized, last_mouse_pos, 5, (0, 0, 0), -1)
        cv2.imshow("colorized depth", depth_colorized)

        if cv2.waitKey(1) == ord('q'):
            break

Thanks !

Antoine

  • erik replied to this.

    Hi apirrone ,
    We apologize for the inconvenience, but the full TOF decoding is not yet implemented, and we are working on it at the moment. It should be ready in firmware in a few week, so you'll be able to just update the library (pip install depthai -U) and have accurate depth.
    Kind regards, Erik

      Thanks erik for the reply, please keep me updated when the update is ready!

      Just curious, how is it that it "kind of works" right now, but with bad accuracy? The colorized depth map looks really good and continuous 😅

      • erik replied to this.

        Hi apirrone ,
        That's the thing - we get some data out, but full decoding and error correction aren't implemented. The depth does look decent (at least the corners are very sharp), but if you'd visualize the pointcloud it's completely off.

        18 days later

        Hi @apirrone ,
        We got the decoding mostly working, there's just some offset left that we haven't figured out. Below is an A4 paper sheet and its size - about 10% dimension mismatch. We believe we'll have it ready this month.
        Thanks, Erik



          18 days later
          9 days later

          Thanks for the update!

          Antoine

          5 days later

          jakaskerl

          hey any update on the ToF sensor finalization so its accurate and ready to be used in depthai?

          jakaskerl

          Also do you have the instructions for how to calibrate the ToF sensor? Do you calibrate the ToF sensor with each camera in the setup?

          ToF is already calibrated, the only thing that's missing is the decoding of the ToF results. Current estimations are end of the month, we apologize for the inconvenience.

          22 days later
          a month later

          Hi @apirrone @tetsuro_okuyama ,
          Sorry about the delay - we have had a bunch of challenges, and are now working on 3 different routines to get this decoding done asap (contractor, vendor, and in-house). We just got some promising results last week, and I think we'll be able to share initial lib version (and some pointclouds) in a week or two. More info to come, we'll likely make a blog post about it.
          Thanks, Erik

          12 days later

          erik

          is the code for this available anywhere? it would be a nice test and a quick demo of its accuracy once it's ready, as we all have A4 sized paper

            Nearpoint

            maybe the demo script could adjusted to recognize a quarter highlight it and provide its x, y, and z dimensions as a demo anyone can quickly run with the tof camera once calibrated.

            10 days later

            @erik any update on the progress for completing the ToF camera code so the accuracy is fixed and we can begin utilizing the component. The ToF camera integration into our product is on hold since the software you guys are working on isnt complete causing invalid accuracy. But once I get word that the software is complete I will be able to start the integration! Would be exciting to start working with the ToF module this week, fingers crossed, lmk! thank u!