• DepthAI

  • ToF sensor: frame rate and sync with stereo stream

Hello,

I'm working with OAK-D SR with ToF and I'm considering the ToF sensor.

# Configure the ToF node

tofConfig = tof.initialConfig.get()

tofConfig.enableOpticalCorrection = True

tofConfig.enablePhaseShuffleTemporalFilter = True

tofConfig.enableTemperatureCorrection = False # Not yet supported

tofConfig.enableFPPNCorrection = True

tofConfig.enablePhaseUnwrapping = True

tofConfig.phaseUnwrappingLevel = 3

tofConfig.phaseUnwrapErrorThreshold = 300

tofConfig.enableBurstMode = False

tofConfig.median = dai.MedianFilter.KERNEL_5x5

tof.initialConfig.set(tofConfig)

tof.setNumShaves(4)

ToF size is 640x480, RGB size 1200x800.

The RGB sensor is set to 30 fps, the ToF sensor to 60 fps.

When I try do get the different streams, I got the following frame rates:

depth: 58 fps

depth + amplitude: 33 fps

depth + intensity: 40 fps

depth + amplitude + intensity: 27 fps

depth + RGB: 17.5 fps

depth + RGB aligned: 10.5 fps

depth + RGB (setIspScale(1, 2)): 30 fps

depth + RGB (setIspScale(1, 2)) aligned: 30 fps

I have two questions:

  1. Are these data consistent with yours or am I missing something?
  2. I am not able to sync the ToF stream and the stereo stream. Can you please share a snippet?

Many thanks!

Best

Alessandro

Hi @AlessandroB ,
At 800P the ImgAlign node is upscaling depth map to match the two. This also increases data needed to send to the host, which (significantly) increases CPU consumption, which hits 100% and everything slows down, that's why you got 10fps. Try using align.setOutputSize(640, 400), I can get 30fps (full code here). This reduces CPU consumption to 75% (also using videoEnc for color stream):

Seems like it's in sync, did you config the Sync node correctly?

Color 1 day, 0:04:30.746224 exposure time 0:00:00.019992
ToF 1 day, 0:04:30.747382 exposure time 0:00:00.000016
----
Color 1 day, 0:04:30.779558 exposure time 0:00:00.019992
ToF 1 day, 0:04:30.780714 exposure time 0:00:00.000016
----
Color 1 day, 0:04:30.812891 exposure time 0:00:00.019992
ToF 1 day, 0:04:30.814048 exposure time 0:00:00.000016

    Hi erik,

    Thank you for your reply and for the snippet.

    Three more questions:

    1. Is it correct to state that align.setOutputSize(640, 400) has a very similar effect to camRgb.setIspScale(1, 2)?
    2. In your answer you mentioned "videoEnc for color stream", but I didn't get when it is used.
    3. Why is it necessary to set SetBlocking to false to the sync node inputs? Must it be applied only to one input or to all but one?

    As regards the stereo-TOF code, I understood where I was wrong. Perhaps I'll have more questions on this topic in the future, but I need to dig deeper by myself before.

    Thanks!

    Alessandro

    • erik replied to this.

        AlessandroB

        1. One will specify depth output size, second will downscale ISP by 1/2 - and depth gets aligned to isp, so depth output would be 400p in both cases.
        2. I didn't use it in gist, I just mentioned that you can use it to reduce CPU by 20%, as you lower bandwidth
        3. setBlocking(False) means it won't block if there are too many frames (it will discard oldest frame). If both streams are at same FPS (and aligned) this setting won't make a difference, as input queue won't fill up anyways

            erik,

            As regards the third point: can it make sense to apply blocking false to all inputs but one?

            This is like we take one of them as a sort of master and then we get the "best" frames for the other inputs. Is rhis interpretation correct or am I wrong?

            Cheers!

            • erik replied to this.