Hi,

I have noticed there is some latency (~80 ms) on my IMU streams (with respect to the latest RGB frame time) when I use an OAK-D Wide that has an BMI270 IMU. But when I test the same code using an OAK-D IOT with an BNO086, the IMU times are properly ahead of the RGB times.

Is there a way to reduce the latency when using the BMI270?

Thanks,
Ibai

Code I used for testing:

#!/usr/bin/env python3

import cv2
import depthai as dai


def timeDeltaToMilliS(delta) -> float:
    return delta.total_seconds() * 1000

# Create pipeline
pipeline = dai.Pipeline()

# Define sources and outputs
imu = pipeline.create(dai.node.IMU)
imu.enableIMUSensor(dai.IMUSensor.ACCELEROMETER_RAW, 400)
imu.enableIMUSensor(dai.IMUSensor.GYROSCOPE_RAW, 400)
imu.setBatchReportThreshold(10)
imu.setMaxBatchReports(10)

xlinkOut = pipeline.create(dai.node.XLinkOut)
xlinkOut.setStreamName("imu")
imu.out.link(xlinkOut.input)

camRgb = pipeline.create(dai.node.ColorCamera)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
camRgb.setInterleaved(False)
camRgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.RGB)

xoutRgb = pipeline.create(dai.node.XLinkOut)
xoutRgb.setStreamName("rgb")
camRgb.isp.link(xoutRgb.input)

monoLeft = pipeline.create(dai.node.MonoCamera)
monoRight = pipeline.create(dai.node.MonoCamera)
depth = pipeline.create(dai.node.StereoDepth)
monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
monoLeft.setCamera("left")
monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
monoRight.setCamera("right")

depth.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.HIGH_DENSITY)
depth.initialConfig.setMedianFilter(dai.MedianFilter.KERNEL_7x7)
depth.setLeftRightCheck(True)
monoLeft.out.link(depth.left)
monoRight.out.link(depth.right)

xout = pipeline.create(dai.node.XLinkOut)
xout.setStreamName("disparity")
depth.disparity.link(xout.input)

# Pipeline is defined, now we can connect to the device
with dai.Device(pipeline, dai.UsbSpeed.SUPER_PLUS) as device:

    # Output queue for imu bulk packets
    imuQueue = device.getOutputQueue(name="imu", maxSize=50, blocking=False)
    rgbQueue = device.getOutputQueue(name="rgb", maxSize=4, blocking=False)
    dispQueue = device.getOutputQueue(name="disparity", maxSize=4, blocking=False)

    imuTs = None
    rgbTs = None
    while True:

        if imuQueue.has():
            imuData = imuQueue.get()  # blocking call, will wait until a new data has arrived
            imuPackets = imuData.packets[-1]
            imuTs = imuPackets.acceleroMeter.getTimestampDevice()

        if rgbQueue.has():
            rgbData = rgbQueue.get()  # blocking call, will wait until a new data has arrived
            rgbTs = rgbData.getTimestampDevice()

        if imuTs is None or rgbTs is None:
            continue
        latency = timeDeltaToMilliS(rgbTs - imuTs)
        tsF  = "{:.03f}"
        print(f"Latency: {tsF.format(latency)} ms")

        if cv2.waitKey(1) == ord('q'):
            break

Hi IbaiGorordo
I tested the latency with the code you supplied above. When using the latest version (2.22) I get latency around 70ms, where IMU is ahead of RGB.
When testing with 2.20.2.0, I get latency ranging from -50ms to 16ms. I believe this is the opposite from what you are experiencing?

Thanks,
Jaka

      jakaskerl

      Thanks for the quick reply!

      I get similar values. But because I set the latency to rgbTs - imuTs, a positive latency would mean that the IMU times are behind.

          Hi IbaiGorordo
          I believe a positive latency would mean the rgbTs is bigger than imuTs, which would mean IMU came before RGB, right?
          Or am I making a mistake here?

          Thanks,
          Jaka

              Hi jakaskerl

              I have created a simple diagram to try to explain. In this context, at a current time (tnow), if we read the latest sensor data (tRGB and tIMU), the stream with the higher timestamp will be the fastest one (closest to tnow). Because of the latency, the IMU sample with a timestamp >= tRGB will take some extra time to arrive.

              Actually, the IMU latency is going to be bigger than what I print in my code, because it doesn't consider the RGB latency.

              • erik replied to this.