Hi y'all! We have used multiple different cameras and boards (FFC3/FFC4, Oak-1, …) mainly with the 9-axis IMU that is onboard. Our use-case are cameras on small to medium-sized boats. We experience permanent unpredictable movement from waves hitting the boat, the boat accelerating promptly. For example, in the following video, we want to get the horizon based on the orientation vector and mapping it to pixel space, but it can't cope with the acceleration: https://youtu.be/fiNu3CDo1lc Hence, I was wondering if there is an option for changing the intrinsic sensor fusion module? Are there different modi/drivers/… that could be changed? So far, doing stuff on the host-side was not too successful as we run into issues with sample rate and synchronization (we work in high-level Python). Generally, it's best to do this as low-level as possible. Could you give me some advice on what to do in this case? Thanks in advance!

IMU Calibration/Sensor Fusion for Dynamic Environment

BenjaminKiefer

Hi y'all!

We have used multiple different cameras and boards (FFC3/FFC4, Oak-1, …) mainly with the 9-axis IMU that is onboard.

Our use-case are cameras on small to medium-sized boats. We experience permanent unpredictable movement from waves hitting the boat, the boat accelerating promptly. For example, in the following video, we want to get the horizon based on the orientation vector and mapping it to pixel space, but it can't cope with the acceleration:

Hence, I was wondering if there is an option for changing the intrinsic sensor fusion module? Are there different modi/drivers/… that could be changed?

So far, doing stuff on the host-side was not too successful as we run into issues with sample rate and synchronization (we work in high-level Python). Generally, it's best to do this as low-level as possible.

Could you give me some advice on what to do in this case?

Thanks in advance!

jakaskerl

BenjaminKiefer Hence, I was wondering if there is an option for changing the intrinsic sensor fusion module? Are there different modi/drivers/… that could be changed?

Not really sure what you mean by that.

Try fusing IMU with feature tracker, example: https://docs.luxonis.com/projects/api/en/latest/samples/FeatureTracker/feature_motion_estimation/

Thanks,
Jaka

BenjaminKiefer

In order to get the orientation vector, you have to fuse accelerometer and gyroscope (and potentially magnetometer) measurements (either through Kalman, Madgwick or similar). However, there are various hyperparameters one can set.

I wondered whether there is any way these are exposed via the Luxonis API so that we can adapt them to different use-cases. As shown in the video, the current settings are not really useful.

Thanks for the suggestion of the feature tracker. The problem in our domain is that the features are pretty bad most of the time (water moves, often time no land visible). Hence, we wanted to rely on an IMU only. We also did some tests with fusing the IMU and vision, but no success so far.

Maybe, it's an inherent weakness of the sensor, but I was just curious to hear if there's anything else that can be done.

jakaskerl

Hi @BenjaminKiefer
Unfortunately there are no ways to change the hyperparameters, they are not exposed on the IMU.
You can try playing around with some different outputs https://docs.luxonis.com/projects/api/en/latest/components/nodes/imu/#imu-sensors. You can also get the raw outputs and do the implementation yourself, but you will need C++ and optimized code to achieve best results.

There is always an option to get a tripod mount with a gyro integrated as well.

Thanks,
Jaka

BenjaminKiefer

Ok, thank you! I'll see what I can do.