Hi from Pollen Robotics :) We just received our ToF module. Using [this example](https://github.com/luxonis/depthai-python/blob/main/examples/ToF/tof_depth.py) we get a pretty nice looking depthmap [upl-image-preview url=https://discuss.luxonis.com/assets/files/2023-10-18/1697632040-137414-capture-decran-du-2023-10-18-14-27-14.png] (Except for the looping colors, but I assume that over a few meters the sensor is not accurate anymore) (We are using an OAK FFC-4P btw) However, when I try to mesure distances, they are pretty far off most of the time. For example Real distance (black circle) \~= 380 [upl-image-preview url=https://discuss.luxonis.com/assets/files/2023-10-18/1697635664-666344-capture-decran-du-2023-10-18-15-23-55.png] Real distance \~= 340 [upl-image-preview url=https://discuss.luxonis.com/assets/files/2023-10-18/1697635721-65842-capture-decran-du-2023-10-18-15-25-52.png] Real distance \~= 195 [upl-image-preview url=https://discuss.luxonis.com/assets/files/2023-10-18/1697635742-552214-capture-decran-du-2023-10-18-15-26-09.png] Real distance \~= 880 [upl-image-preview url=https://discuss.luxonis.com/assets/files/2023-10-18/1697635754-323551-capture-decran-du-2023-10-18-15-27-03.png] A few questions : * I don't understand what the different parameters are doing, and the [documentation](https://docs.luxonis.com/projects/api/en/latest/components/messages/tof_config/#tofconfig) is not very helpful to me (this is the first time I work with a ToF sensor). Could you explain what **freqModUsed, avgPhaseShuffle, and minimumAmplitude** are doing ? * Am I right to expect millimeters out of the depth map ? * Is there any calibration needed ? Could that have been degraded during transport ? Here is my code for depth mesurement in the image (at the position of the cursor), based on the provided example ``` #!/usr/bin/env python3 import cv2 import depthai as dai import numpy as np pipeline = dai.Pipeline() cam_a = pipeline.create(dai.node.Camera) cam_a.setBoardSocket(dai.CameraBoardSocket.CAM_A) tof = pipeline.create(dai.node.ToF) # Configure the ToF node tofConfig = tof.initialConfig.get() tofConfig.depthParams.freqModUsed = dai.RawToFConfig.DepthParams.TypeFMod.MAX tofConfig.depthParams.avgPhaseShuffle = False tofConfig.depthParams.minimumAmplitude = 3.0 tof.initialConfig.set(tofConfig) cam_a.raw.link(tof.input) xout = pipeline.create(dai.node.XLinkOut) xout.setStreamName("depth") tof.depth.link(xout.input) last_mouse_pos = (0, 0) last_depth_mesurement = 0 def mouse_callback(event, x, y, flags, params): global last_mouse_pos last_mouse_pos = (x, y) cv2.namedWindow("colorized depth") cv2.setMouseCallback("colorized depth", mouse_callback) time_averaged_depth = [] # Connect to device and start pipeline with dai.Device(pipeline) as device: print('Connected cameras:', device.getConnectedCameraFeatures()) q = device.getOutputQueue(name="depth") while True: imgFrame = q.get() depth_map = imgFrame.getFrame() depth_downscaled = depth_map[::4] non_zero_depth = depth_downscaled[depth_downscaled != 0] # Remove invalid depth values if len(non_zero_depth) == 0: min_depth, max_depth = 0, 0 else: min_depth = np.percentile(non_zero_depth, 3) max_depth = np.percentile(non_zero_depth, 97) last_depth_mesurement = depth_map[last_mouse_pos[1]-5:last_mouse_pos[1]+5, last_mouse_pos[0]-5:last_mouse_pos[0]+5] last_depth_mesurement = np.mean(last_depth_mesurement) time_averaged_depth.append(last_depth_mesurement) time_averaged_depth = time_averaged_depth[20:] if len(time_averaged_depth) > 20 else time_averaged_depth print("mean :", np.mean(time_averaged_depth), "(std :", np.std(time_averaged_depth), ")") depth_colorized = np.interp(depth_map, (min_depth, max_depth), (0, 255)).astype(np.uint8) depth_colorized = cv2.applyColorMap(depth_colorized, cv2.COLORMAP_JET) depth_colorized = cv2.circle(depth_colorized, last_mouse_pos, 5, (0, 0, 0), -1) cv2.imshow("colorized depth", depth_colorized) if cv2.waitKey(1) == ord('q'): break ``` Thanks ! Antoine

Hi @"apirrone"#p13338 , We apologize for the inconvenience, but the full TOF decoding is not yet implemented, and we are working on it at the moment. It should be ready in firmware in a few week, so you'll be able to just update the library (pip install depthai -U) and have accurate depth. Kind regards, Erik

Thanks @"erik"#p13367 for the reply, please keep me updated when the update is ready! Just curious, how is it that it "kind of works" right now, but with bad accuracy? The colorized depth map looks really good and continuous 😅

Hi @"apirrone"#p13369 , That's the thing - we get _some_ data out, but full decoding and error correction aren't implemented. The depth does look decent (at least the corners are very sharp), but if you'd visualize the pointcloud it's completely off.

Hi @"erik"#175, Any news on the firmware ? Thanks ! Antoine

Hi @"apirrone"#1318 , We got the decoding mostly working, there's just some offset left that we haven't figured out. Below is an A4 paper sheet and its size - about 10% dimension mismatch. We believe we'll have it ready this month. Thanks, Erik [upl-image-preview url=https://discuss.luxonis.com/assets/files/2023-11-06/1699279674-367947-a4-take3.png] [upl-image-preview url=https://discuss.luxonis.com/assets/files/2023-11-06/1699279674-443357-a4-take2.png] [upl-image-preview url=https://discuss.luxonis.com/assets/files/2023-11-06/1699279674-578178-a4-take1.png]

Hi @"erik"#175, any news ? We are very eager to try it out ! :) Thanks, Antoine

Hi @"apirrone"#p14695 We need to finalize a few things, I'd say roughly about 2 weeks.. Thanks, Jaka

@"jakaskerl"#p14717 hey any update on the ToF sensor finalization so its accurate and ready to be used in depthai?

ToF bad accuracy

apirrone

apirrone Sorry I posted too fast, I forgot to checkout develop 🙂 I'm reinstalling it and testing again

apirrone

Ok, so now I'm really on develop, the hash matches the one you sent 2.28.0.0.dev0+76294530fd0bf91b09469490a1d14871c374630a

And I still have the same error

apirrone

I reinstalled everything from scratch, and I don't get the error anymore, must have made a mistake the time before.

Now I have a different problem, the script freezes after a few frames if I try to get a rgb cam and the tof at the same time. If I comment out the rgb queue get part, the for works fine. But I have to comment everything about the ToF (starting at the tof node creation) for the rgb to work.

Here is a MRE :

#!/usr/bin/env python3

import time
import cv2
import depthai as dai
import numpy as np

print(dai.__version__)

cvColorMap = cv2.applyColorMap(np.arange(256, dtype=np.uint8), cv2.COLORMAP_JET)
cvColorMap[0] = [0, 0, 0]


def create_pipeline():
    pipeline = dai.Pipeline()

    tof = pipeline.create(dai.node.ToF)

    # Configure the ToF node
    tofConfig = tof.initialConfig.get()

    # Optional. Best accuracy, but adds motion blur.
    # see ToF node docs on how to reduce/eliminate motion blur.
    tofConfig.enableOpticalCorrection = False
    tofConfig.enablePhaseShuffleTemporalFilter = True
    tofConfig.phaseUnwrappingLevel = 4
    tofConfig.phaseUnwrapErrorThreshold = 300
    tofConfig.enableTemperatureCorrection = False  # Not yet supported

    xinTofConfig = pipeline.create(dai.node.XLinkIn)
    xinTofConfig.setStreamName("tofConfig")
    xinTofConfig.out.link(tof.inputConfig)

    tof.initialConfig.set(tofConfig)

    cam_tof = pipeline.create(dai.node.Camera)
    cam_tof.setFps(30)  # ToF node will produce depth frames at /2 of this rate
    cam_tof.setBoardSocket(dai.CameraBoardSocket.CAM_D)
    cam_tof.raw.link(tof.input)

    xout = pipeline.create(dai.node.XLinkOut)
    xout.setStreamName("depth")
    tof.depth.link(xout.input)

    tofConfig = tof.initialConfig.get()

    left = pipeline.create(dai.node.ColorCamera)
    left.setBoardSocket(dai.CameraBoardSocket.CAM_C)
    left.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1440X1080)
    left.setFps(30)
    left.setIspScale(2, 3)

    left_xout = pipeline.create(dai.node.XLinkOut)
    left_xout.setStreamName("left")
    left.video.link(left_xout.input)

    return pipeline, tofConfig


if __name__ == "__main__":
    pipeline, tofConfig = create_pipeline()

    with dai.Device(pipeline) as device:
        print("Connected cameras:", device.getConnectedCameraFeatures())

        # qDepth = device.getOutputQueue(name="depth")
        qDepth = device.getOutputQueue(name="depth", maxSize=8, blocking=False)

        # left_q = device.getOutputQueue(name="left")
        left_q = device.getOutputQueue(name="left", maxSize=8, blocking=False)

        while True:
            start = time.time()

            imgFrame = qDepth.get()  # blocking call, will wait until a new data has arrived
            depth_map = imgFrame.getFrame()

            max_depth = (tofConfig.phaseUnwrappingLevel + 1) * 1500  # 100MHz modulation freq.
            depth_colorized = np.interp(depth_map, (0, max_depth), (0, 255)).astype(np.uint8)
            depth_colorized = cv2.applyColorMap(depth_colorized, cvColorMap)

            # If I comment that (3 next lines), the tof works fine
            in_left = left_q.get()
            left_im = in_left.getCvFrame()
            cv2.imshow("left", left_im)
            # Until that

            cv2.imshow("Colorized depth", depth_colorized)
            key = cv2.waitKey(1)

    device.close()

Luxonis-Alex

@apirrone That's great that the I/O error is not showing up, but have been something with the build (maybe submodules not updated). Just mentioning, a way to auto-update submodules and install our CI prebuilts (if available) is to run from depthai-python:
python3 examples/install_requirements.py
Or specifically a version:
python3 -m pip install --extra-index-url https://artifacts.luxonis.com/artifactory/luxonis-python-snapshot-local depthai==2.28.0.0.dev0+76294530fd0bf91b09469490a1d14871c374630a

Now for the freeze issue you're observing, I noticed the same myself when the extra 5V power was not applied, IMX296 started streaming and after a few frames it froze, when ToF started. Was fixed with the DC jack, but I also have only a single IMX296 RPi GSCam module locally to test with (so not sure if it could still be a problem with 2x IMX296 + ToF):

With all 3 cameras attached, can you check if enabling just a few of them is fine, for example with cam_test:
python3 utilities/cam_test.py -cams camb,c camc,c
python3 utilities/cam_test.py -cams camb,c camd,t
python3 utilities/cam_test.py -cams camc,c camd,t
python3 utilities/cam_test.py -cams camb,c camc,c camd,t

apirrone

Hi @Luxonis-Alex

Plugging the 5V external power supply did not change anything. I also tried unplugging one of the IMX296, same behavior, the whole pipeline freezes after a few frames (like 5 frames).

No problem getting the two IMX296 alone, and the ToF alone.

Luxonis-Alex

@apirrone We can try to replicate the problem, can you check the revisions of all boards (the FFC-4P DD2090 RxMx.., RPi camera connector adapter, ToF module...).
A picture of your setup would also be helpful. Thanks!

apirrone

can you check the revisions of all boards (the FFC-4P DD2090 RxMx.., RPi camera connector adapter, ToF module...).

Can you guide me on how to get all this information ? Using device_manager.py I get that

Meanwhile, here are some photos of my setup

apirrone

Also, not exactly related question, but in order to use the align node, I know the tof needs to be aligned with the rgb camera horizontally, but how about in the z direction ? (away from the camera) Is it critical that their focal planes are aligned for example ? If so, how do I know how to align them in this direction ?
Thanks !

Luxonis-Alex

@apirrone We'll follow up on this question about alignment.

For the issue with IMX296 RPi GS Cam not working together with the ToF, would you be able to try a rework on the SL6996 R1M1E1 cable adapter boards, it would be replacing this 10K resistor (the populated one) with a 1K resistor. Or if easier, just soldering a 1K resistor in parallel with the existing 10K one (on top of it), final value isn't that important:

Design files of that adapter: luxonis/depthai-hardwaretree/master/SL6996_OAK-FFC_15pin-RPi

apirrone

Luxonis-Alex I'll try that right now

apirrone

It was hard, but I managed to solder a 1k resistor this way 🙂

And ... it works ! I no longer have the freeze ! I only tested with one camera for now, I'll try soldering another resistor to the other board to see how it behaves with the two cameras

Thanks @Luxonis-Alex ! Can you explain what was going on ?

Hester

apirrone use surface mount resistor

apirrone

jakaskerl

So If I understand correctly, I should set the camera extrinsics as follows for example ? :

right->left
left->tof
tof->-1

I get an error if I set -1 or "-1" for toCameraSocket

Traceback (most recent call last):
  File "/home/antoine/Pollen/pollen-vision/pollen_vision/pollen_vision/camera_wrappers/depthai/calibration/flash.py", line 31, in <module>
    w.flash(args.calib_json_file, tof=args.tof)
  File "/home/antoine/Pollen/pollen-vision/pollen_vision/pollen_vision/camera_wrappers/depthai/wrapper.py", line 312, in flash
    ch.setCameraExtrinsics(
TypeError: setCameraExtrinsics(): incompatible function arguments. The following argument types are supported:
    1. (self: depthai.CalibrationHandler, srcCameraId: depthai.CameraBoardSocket, destCameraId: depthai.CameraBoardSocket, rotationMatrix: list[list[float]], translation: list[float], specTranslation: list[float] = [0.0, 0.0, 0.0]) -> None

Invoked with: <depthai.CalibrationHandler object at 0x7e4bb400a5f0>, <CameraBoardSocket.CAM_D: 3>, '-1', [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]], [-3.2509, 0.0, 0.0]; kwargs: specTranslation=[-3.2509, 0.0, 0.0]

So, should it look like this ?

ch.setCameraExtrinsics(
    left_socket,
    tof_socket,
    R_left_to_tof,
    T_left_to_tof,
    specTranslation=T_left_to_tof,
)

ch.setCameraExtrinsics(
    right_socket,
    left_socket,
    R_right_to_left,
    T_right_to_left,
    specTranslation=T_right_to_left,
)

# Should I do this ?
ch.setCameraExtrinsics(
    tof_socket,
    tof_socket,
    [0, 0, 0],
    np.eye(3).tolist(),
    specTranslation=[0, 0, 0],
)

jakaskerl

apirrone

apirrone I get an error if I set -1 or "-1" for toCameraSocket

You have to set it to CameraBoardSocket.AUTO instead of CameraBoardSocket.CAM_X. The enum value is -1.

apirrone How can I get the real intrinsic matrix of the ToF ? I guess it's flashed in it's own eeprom ?

Yes, the intrinsics are flashed in the module eeprom.

apirrone Is there an easy way to get the depth in full resolution and the relevant aligned rgb section cropped ? So that I have two 640x480 (depth and rgb) that are aligned ?

Undistort the RGB image first. More on this here.

Thanks,
Jaka

apirrone

jakaskerl Make sure the .json in properly flashed. I find that using your config the camera on socket 3 would not get flashed, if I didn't specify the width and height.

Indeed, I have to set camera intrinsics with width and height, then the align node works !

tof-2024-08-27-144757.mp4

6MB

As you can see, it's not very well aligned for now, that's because I set the tof's K matrix to this :

        if tof:
            K = np.eye(3)
            K[0][0] = 640  # fx
            K[1][1] = 640  # fy
            K[0][2] = 320  # cx
            K[1][2] = 240  # cy
            tof_socket = get_socket_from_name("tof", self.cam_config.name_to_socket)
            ch.setCameraIntrinsics(tof_socket, K.tolist(), (640, 480))

How can I get the real intrinsic matrix of the ToF ? I guess it's flashed in it's own eeprom ?

apirrone

Also, we are using very wide FOV lenses, as you can see below

Is there an easy way to get the depth in full resolution and the relevant aligned rgb section cropped ? So that I have two 640x480 (depth and rgb) that are aligned ?

apirrone

jakaskerl Yes, the intrinsics are flashed in the module eeprom.

So no way to get them ? Do I need them to align the tof to RGB ?

I needed to set the camera intrinsics in the calibration handler for it to flash properly, but you said I had to set the size of the image, is there a way to set the image size without setting the intrinsics ?

How do you flash the calibration for the SR-PoE-ToF ?

jakaskerl You have to set it to CameraBoardSocket.AUTO instead of CameraBoardSocket.CAM_X. The enum value is -1.

In the end I set tof->left, left->tof and left->right, this satisfies this :

jakaskerl No two sockets can have the same toCameraSocket value

And seems to be enough

apirrone

jakaskerl Undistort the RGB image first. More on this here.

If I undistort the RGB image first, this is what I get :

No undistort :

Undistort:

apirrone

jakaskerl This script has never worked for me, that's why I made a custom calibration process.

I tried again still, in the tof_calib branch, with a fresh virtual environment and I have to iteratively comment out / edit / add things in the script to move on with the process. For example, for some reason, these parameters don't exist in this branch


                    tof_config.enablePhaseShuffleTemporalFilter = True
                    tof_config.enableFPPNCorrection = True
                    tof_config.enableOpticalCorrection = True
                    tof_config.enableWiggleCorrection = True
                    tof_config.enableTemperatureCorrection = True
                    tof_config.phaseUnwrappingLevel = 4
                    tof_config.phaseUnwrapErrorThreshold = 1000

while they exist here 2.28.0.0.dev0+76294530fd0bf91b09469490a1d14871c374630a.

Same for intensity, which means I'm stuck here luxonis/depthaiblob/28f2bd654923e24019f00248cf03925866b107af/calibrate.py#L608

So I recorded the images and computed the calibration using my usual process and I still have alignement issues.

This is where I flash the calibration : pollen-robotics/pollen-visionblob/01f1901a95f44e052b2953e4453247c6d7aa6b47/pollen_vision/pollen_vision/camera_wrappers/depthai/wrapper.py#L260

Now I get the intrinsics and extrinsics from my calibration process
It seems better than before :

But there is still a misalign when my arm is closer for example

Now, when I undistort the rgb first

jakaskerl Undistort the RGB image first. More on this here.

If doesn't change much because most of the distortion is on the edge of the FOV and the ToF's FOV is is much smaller than our fisheye lenses.

Also, the ToF can get pretty noisy sometimes (as you can see on top of the image below):

Do you have any advice ?

Thanks

jakaskerl

apirrone
Looks better, but the alignment is still off and the scale is wrong.
Can you post the calibration json as well?

Thanks,
Jaka

apirrone

Hester I tried, but it's so small I couldn't solder it properly 🙂

Hester

apirrone There are different sizes

« Previous Page Next Page »