Hey guys,

Thanks for the nice products, I really like it and its SDK.

However, I want to deploy the Oak-d pro wide in a setting where I have a lot of jitter and general movement (on a boat).

Therefore, I want to do some video stabilization. I tested with some basic LK optical flow methods, but I do not get good results. Purely IMU-based methods fail in many situations (boat is moving fast and a lot of waves).

Before I'm going down the fusion route (where I don't have much experience unfortunately), I wondered if there already is some existing code for that. I could not find anything on that other than the "smoothed" IMU rotation vector.

What would you recommend doing going further?

Thanks!

    Hi BenjaminKiefer
    I don't think we have any existing experiments featuring a fusion of IMU and camera. Optical flow methods would work better if there wasn't so much high frequency jitter. Probably the easiest way would be to mount a camera on a stabilizer and doing so filter out the jitter. After that you can apply optical flow for additional image stabilization. It should work much better.

    Thoughts?
    Jaka

      Thanks for your suggestion!

      What kind of stabilizer did you think of? Do you have any suggestion what could be used?

      The only stabilizers I know are for big cameras and it's hard to mount that. Plus, I'm not sure how good it is to have such a moving part on a boat in rough conditions (water, wind, …).

      I know that cameras, such as the Insta360 have really solid stabilization (although not real-time).

      But besides high-frequency jitter:

      Also the low-frequency ones do not work too well. If I'm going fast on a boat, the IMU is going crazy and if there are not many stable features, also optical-flow-based methods don't perform well. Do you know of any repo that you can suggest?

        Hi BenjaminKiefer
        I'm not too knowledgeable in camera stabilization, but perhaps some smaller stabilizers (like the ones use for mobile phones) will work fine since they are designed for low weight cameras. Ideally you would need very loose springs that would filter out the jitter.

        Perhaps look at some examples of image stabilization that is not based on optical flow methods. Maybe this library helps you achieve better results.
        Script using the library

        For IMU integration, the imu has x, y and z component. I would assume the component in the forward direction vector should be 0, the same should be for left/right (most of the time driving forward). What changes is up/down component which should change slow enough for imu to sample it correctly (IMU can easily run at 100+ Hz).

        Thoughts?
        Jaka