TOF FPS issues on DepthAI-v3

Penguinmon

Hello, I am using the TOF node on an OAK TOF device with depthai==3.0.0rc4, and I can't get more than 14 FPS on it. There also seems to be extreme motion blur whenever I stream the depth.

#!/usr/bin/env python3
import time
import depthai as dai

with dai.Pipeline() as pipeline:
    tof = pipeline.create(dai.node.ToF).build(fps=40)
    tof_out = tof.depth.createOutputQueue()

    pipeline.start()

    frame_count = 0
    fps_frames = 0
    t_win = time.perf_counter()

    while pipeline.isRunning():
        inDepth = tof_out.get()

        frame_count += 1
        fps_frames += 1

        # Print FPS once per second
        now = time.perf_counter()
        if now - t_win >= 1.0:
            fps = fps_frames / (now - t_win)
            print(f"FPS: {fps:.2f}")
            fps_frames = 0
            t_win = now

Penguinmon

I could get at least 30 FPS with minimal code when using DepthAI-v2.

Product Name : OAK-D-SR-POE
Bootloader Version: 0.0.28

OskarSonc

Hey @Penguinmon

What you’re seeing is expected — ToF depth frames are built from multiple raw captures, so the effective FPS is lower and moving objects can blur.

You can find recommendations on how to reduce blur and improve FPS here:
https://docs.luxonis.com/software-v3/depthai/depthai-components/nodes/tof/#tof-motion-blur

Thanks
Oskar

Penguinmon

Is the motion blur supposed to be this bad after following those changes? I also can't get more than 8 FPS with burst mode enabled using depthai v3. I don't have any of these issues when using depthai v2.

OskarSonc

Hi @Penguinmon sorry for delay, had to check with the team…

The blur you see is due to temporal filtering in v3. You can try this demo for reference:
https://github.com/lnotspotl/oak-examples/blob/lnotspotl/tof_pointcloud_demo/depth-measurement/3d-measurement/tof-pointcloud/tof.py

In that example you can disable temporal filtering and leave only spatial filtering (runs on host). Keep in mind if the host is slow, FPS can drop. If you want the same “raw” output as in v2, use the rawDepth output of the ToF node.

Hope this helps,
Oskar

Penguinmon

That helped a lot!

When temporal filtering was mentioned, I thought it was the phase shuffle temporal filter, but now I see there is a temporal filter in the image filters config that I should've been tuning.

OskarSonc

Glad it helped 🙂

SimonvE

I had the same issue but I thought this solution would not apply to me as I did not set-up an ImageFilters node. However only now after a lot more research I see "depthai.node.ToF.imageFiltersInputConfig" in the API. Are these filters in place by default? This is hidden very well and not explained to the user anywhere.

Please change the TOF node such that the configuration is clear, with all options laid-out and its functionality documented to the user where the user expects it.

OskarSonc

SimonvE - thanks, will send this to the team, to update the docs.

BionicBoon

OskarSonc Thank you for picking this up.

Adding to the comment of @SimonvE : I have found the ToF documentation to be difficult to comprehend and unravel in general. Not just the ToF Node, but for example also the ToFDepthConfidenceFilter (What is the range of the confidence value? Is higher better or worse? How can we set the threshold? etc.).

Thank you in advance, looking forward to updated docs.

OskarSonc

BionicBoon - thanks, will let you know

SimonvE

OskarSonc Are there any updates on this topic?

Thanks in advance too

OskarSonc

SimonvE sadly no, but I can help you with something if you need...

BionicBoon

OskarSonc Hi Oskar,

I have a question in the meantime (although I would like to reiterate that better docs are really vital for smooth development on our side!).

I have changed up my setup from a dynamic to semi-static one. However, I'm experiencing the opposite issue of Simon in this case: My temporal filtering does not seem to work at all. Moreover, I have a bunch of 'random' noise spots which only last a few frames each time. See the attached video. I'm specifically talking about the (typically red-coloured) 'blobs' which occur near the conveyor belt.

depthnoise-static.mp4

16MB

I have tried changing the temporal settings: Alpha ranging from 0.0 & 0.1 to 0.4 and 0.7, but to no avail. It also doesn't seem like I'm temporally filtering, as I'm observing no 'blur' effect. My code is attached below for initialisation:

class Camera():

"""

Camera class: Overarching Vision class, which connects with the camera and analyses frames for oriented boxes\\n

"""



def __init__(self):

    """

    Constructor method\\n

    :param settings: Settings related to the camera hardware

    :type settings: CameraSettings

    :param RGBFiltSettings: Settings related to the filtering of RGB frames to detect bundle contours

    :type RGBFiltSettings: RGBFilterSettings

    """

    self.camAttached : bool = True

    self.newConfig : bool = True

    devIP = "172.31.1.183"

    deviceInfo = dai.DeviceInfo(devIP)

    self.device = dai.Device(deviceInfo)

    print(f"DepthAI v{dai.__version__}")

    print(f"Connected to camera {self.device.getDeviceName()} at {devIP}")

    print(f"Sensors: {self.device.getCameraSensorNames()}")

    self.camResolution : List[int] = [640,480]

    self.pipeline = dai.Pipeline(defaultDevice=self.device)

    self.remoteConnector = dai.RemoteConnection(address="0.0.0.0",webSocketPort=8765,serveFrontend=True,httpPort=8080)

    self.tofCam = self.pipeline.create(dai.node.ToF).build(dai.CameraBoardSocket.CAM_A)

    self.tofConfig = dai.ToFConfig()

    self.tofConfig.enablePhaseShuffleTemporalFilter = True

    self.tofConfig.setMedianFilter(dai.MedianFilter.KERNEL_7x7)

    self.tofConfig.enablePhaseUnwrapping = True

    self.tofConfig.phaseUnwrappingLevel = 4 # Up to \~ 7.5 m

    self.tofConfig.phaseUnwrapErrorThreshold = 1000 # [mm]

    self.tofConfig.enableOpticalCorrection = True

    self.tofConfig.enableBurstMode = False

    self.tofCam.setInitialConfig(self.tofConfig)        

    self.depthConfigQ = self.tofCam.imageFiltersInputConfig.createInputQueue() # ToF node has built-in post-processing

    self.depthProcessConfig =dai.ImageFiltersConfig()

    temporal = dai.node.ImageFilters.TemporalFilterParams()

    temporal.alpha = 0.1 # Tried variosu values here, e.g. 0.0, 0.4, 0.7, […]

    temporal.delta = 0 # Auto

    temporal.persistencyMode = dai.node.ImageFilters.TemporalFilterParams.PersistencyMode.VALID_8_OUT_OF_8 # Tried various modes here…

    temporal.enable = True

    self.depthProcessConfig.filterParams = [temporal]



    self.rgbCam = self.pipeline.create(dai.node.Camera).build(dai.CameraBoardSocket.CAM_C)

    self.rgbConfig = dai.CameraControl()

    self.rgbConfig.setAntiBandingMode(dai.CameraControl.AntiBandingMode.MAINS_50_HZ)

    self.rgbConfig.setSaturation(3)

    self.rgbConfig.setManualExposure(10000, 100)

    self.rgbConfigQ = self.rgbCam.inputControl.createInputQueue()

    self.rgbStream = self.rgbCam.requestOutput(tuple(self.camResolution), dai.ImgFrame.Type.BGR888p, 

                                               dai.ImgResizeMode.CROP, 15, True)

    self.rgbQ = self.rgbStream.createOutputQueue(maxSize=20, blocking=False)       

    self.aligner = self.pipeline.create(dai.node.ImageAlign)

    self.aligner.setRunOnHost(True)

    self.tofCam.depth.link(self.aligner.input)

    self.rgbStream.link(self.aligner.inputAlignTo)

    self.depthQ = self.aligner.outputAligned.createOutputQueue(maxSize=20, blocking=False)

    self.remoteRGBQ = self.remoteConnector.addTopic(topicName="RGB", group="common",maxSize=2,blocking=False,useVisualizationIfAvailable=True)

    self.remoteRGBDQ = self.remoteConnector.addTopic(topicName="RGBD", group="common",maxSize=2,blocking=False,useVisualizationIfAvailable=True)

    self.remoteDepthQ = self.remoteConnector.addTopic(topicName="Depth", group="common",maxSize=2,blocking=False,useVisualizationIfAvailable=True)

    self.pipeline.start()

    self.remoteConnector.registerPipeline(self.pipeline)

And then at the start of my main thread:

cam = Camera()

try:

    while(cam.pipeline.isRunning()):

        if cam.newConfig:

            cam.rgbConfigQ.send(cam.rgbConfig)

            cam.depthConfigQ.send(cam.depthProcessConfig)

            cam.newConfig = False

        newRGBFrame = cam.GrabRGBFrame()

        newDepthFrame = cam.GrabDepthFrame()

[…]

Any idea what I might be doing wrong?

OskarSonc

Hey @BionicBoon - Setting temporal.delta = 0 is not correct

Temporal Filter

The TemporalFilter is designed for dynamic smoothing of depth data over time. It blends incoming frames with accumulated history, applies a threshold for significant changes, and retains past data using configurable persistence rules. This filter is particularly useful for reducing noise in depth data or temporal inconsistencies in image sequences.

How it works

For each pixel, the filter compares the new value I_new with the accumulated value I_acc:

1) Blend (small change)

If the difference is below the threshold delta, the filter blends the values:

I_out = alpha * I_new + (1 - alpha) * I_acc

2) Replace (large change)

If the difference is greater than or equal to delta, the new pixel replaces the accumulated value:

I_out = I_new

3) Persistence (missing/zero input)

If the new pixel is zero, the persistence rule decides whether to keep history or drop to zero:

I_out = I_acc (if the persistence condition is satisfied)

I_out = 0 (otherwise)

Important note about delta = 0

When delta is 0, any non-identical change is treated as “significant,” so blending almost never happens. In practice, that removes the temporal continuity the filter is meant to provide.

BionicBoon

OskarSonc Thanks for your response. However, you did not provide me with the value you recommend to try? Any tips here, as to what is acceptable / typically useful?

Note, however, that your own documentation informed me that setting the value to zero would automatically tweak it. Therefore I would like to reiterate SimonvE and my own point about updating the documentation…

DavidZah

Understanding ToF Temporal Filter Tuning

Determining exact parameters can be challenging as they are highly dependent on your specific environment. However, based on the sensor characteristics of our ToF modules, here is a technical baseline to help you tune your setup:

1. The "Delta" Parameter

The 2 sigma variance of typical ToF noise is approximately 10 mm.

For General Use: A delta of 15 is usually sufficient to suppress standard depth noise.
For Semi-Static Scenes (e.g., Conveyors): If you are working with slow-moving objects and want to eliminate those "random" noise blobs, I recommend increasing the delta to 30 or even 45. This higher threshold allows the filter to blend small fluctuations while still reacting to actual physical movement.

2. Blending with Alpha

Since you mentioned a lack of "blur" effect, ensure your alpha is set between 0.1 and 0.4. A lower alpha increases the influence of historical frames, leading to smoother data but more potential motion trail (blur).

3. Documentation Update

I sincerely apologize for the confusion caused by the current documentation regarding the "zero" value behavior. We are currently overhaulng the DepthAI v3 ToF documentation to ensure these parameters and their "hidden" defaults are transparently explained.

We appreciate your patience while we improve these resources. Please don't hesitate to reach out here with further questions in the meantime.

BionicBoon

Thanks for the feedback (although the AI-generated answer made it come across a bit 'chaotic').
I tried to play with delta and alpha, setting them to their extremes (e.g. alpha = 0.01 and delta = 200). Although I can notice the 'temporal blurring' on "well-behaved" regions, I still get spotty data at what I suppose are low confidence regions (right side in the video). Does this have something to do with hole filling? I tried playing with Persistency modes, but that didn't solve it.

Kindly advice me on what settings I might change to make these spotty areas more consistent. The area I will be observing is semi-static (can be assumed static over the period that I sample), so we can filter heavily temporally speaking…

screencast-from-23-01-26-09-49-44.webm

185kB

BionicBoon

@OskarSonc @DavidZah any news on this?