Using the IMU to compensate for imperfect mounting of camera

I was thinking I could use the rotation vector from the IMU when the application starts up to get a measurement for how the camera was mounted and later use that to correct spatial measurements relative to the real world.
In my case the camera will move around one axis when used (yaw) so it seems the easiest to just use the IMU to correct for yaw and pitch from startup.
There is a nice description in https://discuss.luxonis.com/d/383-imu-and-camera-coordinate-frames/2 for how the IMU axis are set up.
Thus I would like to ignore the x axis and get the deviation for y and z compared to a perfect camera mount ("x straight down, y straight sideways and z straight forward").
Having just this "yz deviation" in a quaternion would make it very simple to compensate the spatial coordinates I believe (using the inverse of it).
It seems harder than I expected to isolate that yz deviation from the IMU readings. Has anyone tried to do something like this already and succeeded? Or perhaps there is a simpler way to go about it...

Hi erik ,
I was contemplating some kind of calibration alternative, so thanks for pointing me to that example. That seems like the better way to go although it adds an extra manual step.
It means I need to build some contraption to hold the chessboard pattern in a precise way in the world I want, right? I assume I should put the chessboard not too far away from the camera and straight in front of it. I'll need some experimentation to see what kind of transformation I get.
Looks like this calibration code only exists in python, right?