I was trying out the `depthai-python/examples/IMU` on the **Oak D Wide** and found that when I ran the `imu_gyroscope_accelerometer.py` the timestamps of the accelerometer and gyroscope don't seem to sync up: [URL=https://i.imgur.com/6ObVnYf.png][IMG]https://i.imgur.com/6ObVnYfh.png[/IMG][/URL] I happen to have a **Oak D Lite** and confirm the IMU data seems synchronized: [URL=https://i.imgur.com/fK7PhjF.png][IMG]https://i.imgur.com/fK7PhjFh.png[/IMG][/URL]

> The accelerometer is synced with gyro, the combined sync output rate is the minimum between the two (400 hz in this case). Yes, but this is not what I'm seeing with the **Oak D Wide**. The Oak D Wide **accelerometer and gyroscope outputs are not synchronized (in terms of time)**, as I have shown in the screenshots above. This is important because state-of-the art SLAM / VIO algorithms assume at time-step `k` you have the IMU acceleration and angular velocity measurement at that time instance. While the OAK D Lite does satisfy this requirement the **OAK D Wide** does not. Reference: - [On-Manifold Preintegration for Real-Time Visual-Inertial Odometry by Christian Forster, Luca Carlone, Frank Dellaert, Davide Scaramuzza](https://rpg.ifi.uzh.ch/docs/TRO16_forster.pdf) - [A Multi-State Constraint Kalman Filter for Vision-aided Inertial Navigation by Anastasios I. Mourikis and Stergios I. Roumeliotis](https://ieeexplore.ieee.org/document/4209642) > Essentially every 5th accelerometer is discarded to get a synced output w/ gyro. This is where I would disagree technically. You don't have to throw away anything if one does a simple linear interpolation. Is there a way for me to obtain the raw accelerometer and gyroscope measurement data independently? So that I can synchronize / interpolate them myself?

IMU data synchronization issues on the Oak D Wide

chutsu

I was trying out the depthai-python/examples/IMU on the Oak D Wide and found that when I ran the imu_gyroscope_accelerometer.py the timestamps of the accelerometer and gyroscope don't seem to sync up:

I happen to have a Oak D Lite and confirm the IMU data seems synchronized:

chutsu

Doing some more investigation with the IMU measurements, the data rates are very odd to me. Below are the measurement rates which are stamped by the device.

The gyroscope and accelerometer rates vary very wildly on the OAK D WIDE. These rates are basically calculated via 1.0 / (current_time - previous_time).

With the OAK D LITE things are *better per se but still not ideal. I feel the "synchronization" strategy is not ideal...

One last thing, both experiments are configured with the IMU Acc at 500 and IMU Gyro at 400. While the OAK D WIDE does sort of forfil the rates albeit with very wide variances, the OAK D LITE is no where near the defined 500 / 400 Hz for the IMU Acc and Gyro respectively.

Here is the script I used to produce the above:

#!/usr/bin/env python3
import cv2
import numpy as np
import matplotlib.pylab as plt
import depthai as dai


def ts2sec(delta):
  """Timestamp to Seconds"""
  return delta.total_seconds()


if __name__ == "__main__":
  # Create pipeline
  pipeline = dai.Pipeline()

  # Configure IMU
  imu_xout = pipeline.create(dai.node.XLinkOut)
  imu_xout.setStreamName("imu")

  imu = pipeline.create(dai.node.IMU)
  imu.enableIMUSensor(dai.IMUSensor.ACCELEROMETER_RAW, 500)
  imu.enableIMUSensor(dai.IMUSensor.GYROSCOPE_RAW, 400)
  imu.setBatchReportThreshold(1)
  imu.setMaxBatchReports(1)
  imu.out.link(imu_xout.input)

  # Connect to device and start pipeline
  device = dai.Device(pipeline)

  # Loop
  imu_queue = device.getOutputQueue(name="imu", maxSize=50, blocking=True)

  num_measurements = 0
  imu_acc_time = []
  imu_acc_data = []
  imu_gyr_time = []
  imu_gyr_data = []

  loop = True
  while loop:
    imu_data = imu_queue.get()
    for imu_packet in imu_data.packets:
      imu_acc = imu_packet.acceleroMeter
      imu_gyr = imu_packet.gyroscope
      imu_acc_ts = ts2sec(imu_acc.getTimestampDevice())
      imu_gyr_ts = ts2sec(imu_gyr.getTimestampDevice())

      # Keep track of measurements
      imu_acc_time.append(imu_acc_ts)
      imu_gyr_time.append(imu_gyr_ts)
      imu_acc_data.append([imu_acc.x, imu_acc.y, imu_acc.z])
      imu_gyr_data.append([imu_gyr.x, imu_gyr.y, imu_gyr.z])

      # Break loop?
      num_measurements += 1
      if num_measurements > 100:
          loop = False
          break

  # Plot data
  imu_acc_time = np.array(imu_acc_time)
  imu_acc_data = np.array(imu_acc_data)
  imu_gyr_time = np.array(imu_gyr_time)
  imu_gyr_data = np.array(imu_gyr_data)
  print(f"imu rate: {1.0 / np.mean(np.diff(imu_acc_time))}")
  print(f"gyr rate: {1.0 / np.mean(np.diff(imu_gyr_time))}")

  # Boxplot
  plt.figure()
  plt.boxplot([1.0 / np.diff(imu_acc_time), 1.0 / np.diff(imu_gyr_time)])
  plt.xticks([1, 2], ["Accelerometer", "Gyroscope"])
  plt.ylabel("Rate [Hz]")

  # Time series plot
  plt.figure()
  plt.plot(1.0 / np.diff(imu_acc_time), 'r.', label="Acceleromenter")
  plt.plot(1.0 / np.diff(imu_gyr_time), 'b.', label="Gyroscope")
  plt.legend(loc=0)
  plt.xlim([0, num_measurements])
  plt.xlabel("i-th Measurement")
  plt.ylabel("Rate [Hz]")

  plt.show()

jakaskerl

Hi chutsu
The accelerometer is synced with gyro, the combined sync output rate is the minimum between the two (400 hz in this case).
Essentially every 5th accelerometer is discarded to get a synced output w/ gyro.

This means that the way you measure data rate, you get variation in accelerometer samples (because every 5th is discarded). However if you would measure it as samples/second, then it would be more even.

Thanks,
Jaka

chutsu

The accelerometer is synced with gyro, the combined sync output rate is the minimum between the two (400 hz in this case).

Yes, but this is not what I'm seeing with the Oak D Wide. The Oak D Wide accelerometer and gyroscope outputs are not synchronized (in terms of time), as I have shown in the screenshots above. This is important because state-of-the art SLAM / VIO algorithms assume at time-step k you have the IMU acceleration and angular velocity measurement at that time instance. While the OAK D Lite does satisfy this requirement the OAK D Wide does not.

Reference:

Essentially every 5th accelerometer is discarded to get a synced output w/ gyro.

This is where I would disagree technically. You don't have to throw away anything if one does a simple linear interpolation. Is there a way for me to obtain the raw accelerometer and gyroscope measurement data independently? So that I can synchronize / interpolate them myself?

erik

Hi @chutsu ,
We apologize for the late reply, had to check with the engineering team.
You are correct, measurements are not discarded, it's just that gyro and accelerometer are running at different rates, so the only way to have them sycned would be to either have them running at the same rate, or to interpolate between readings.

raw accelerometer and gyroscope measurement data independently?

Could you expand on this - what's currently missing from the API perspective?
Thanks, Erik

chutsu

erik

Could you expand on this - what's currently missing from the API perspective?

From an API perspective I would prefer getting the Accelerometer / Gyroscope measurements independently. i.e. Accelerometer / Gyroscope at 500Hz and 400Hz respectively as I have configured them while enabling the IMU sensors via

imu = pipeline.create(dai.node.IMU) imu.enableIMUSensor(dai.IMUSensor.ACCELEROMETER_RAW, 500) imu.enableIMUSensor(dai.IMUSensor.GYROSCOPE_RAW, 400)

This would allow the developer to obtain the full capability of the sensor themselves (gyro / accel / or both) or interpolate the measurements to how they see fit. From my understanding there is no way to get them independently.

As an example, this is how intel realsense exposes their IMU measurements: link

chutsu

@erik

You are correct, measurements are not discarded, it's just that gyro and accelerometer are running at different rates, so the only way to have them sycned would be to either have them running at the same rate, or to interpolate between readings.

Yes, and I have shown in the screenshot above the OAK-D Wide accel / gyroscope measurements are not synchronized since their timestamps do not align.

ShivaRamoudith

@chutsu Were you able to resolve this issue beyond just doing interpolation?

I'm essentially encountering this same problem as I'm using this data for VIO work - I need to have the accelerometer and gyroscope readings separately because of the difference in the stable rates of the accelerometer (250 Hz) and gyroscope (200 Hz) on my OAK-D. @jakaskerl

jakaskerl

ShivaRamoudith
You don't have other choice but to interpolate. The depthai-ros has this functionality already integrated into the driver. Unfortunately this is the limitation of the IMU.

Thanks,
Jaka