enableUndistortion has no effect anymore

Seneral

I am suddenly unable to get undistorted images from my OAK-D Lite camera.
It did work last time I worked with it 3 months ago, and I don't think I even updated the depthai package since then (I did test both 3.5.0 and 3.7.1 just in case, to no avail).
Facts I confirmed:
a) undistortion worked previously
b) enableUndistortion has no effect now, camera image is distorted
c) manually undistorting using OpenCV with getTransformation coeffs does result in a correctly undistorted image

I verified b) not just in my main script but also the example script I found here changed to Full HD Color Camera:

#!/usr/bin/env python3

import cv2
import depthai as dai

with dai.Pipeline() as pipeline:
    cam = pipeline.create(dai.node.Camera).build(dai.CameraBoardSocket.CAM_A)
    croppedQueue = cam.requestOutput((1920,1080), resizeMode=dai.ImgResizeMode.CROP, enableUndistortion=True).createOutputQueue()
    distortedQueue = cam.requestOutput((1920,1080), enableUndistortion=False).createOutputQueue()

    pipeline.start()
    while pipeline.isRunning():
        cv2.imshow("cropped undistorted", croppedQueue.get().getCvFrame())
        cv2.imshow("distorted", distortedQueue.get().getCvFrame())
        if cv2.waitKey(1) == ord("q"):  break
    pipeline.stop()

Here's an image to corroborate that the above scripts output are both the same (showing the upper edge of a monitor aligned with the upper edge of the image:

Only the manually undistorted one (top) is actually straight, both outputs directly from the camera (below) are the same.
Any idea why this could be happening? Perhaps there was a regression and I just didn't realise until now?

Oskar-Luxonis

Hey Seneral - This may be caused by the calibration EEPROM format/flags rather than the enableUndistortion call itself.

Could you check these fields on your device and send it?

import depthai as dai

with dai.Device() as dev:
    eeprom = dev.readCalibration().eepromToJson()
    print("version:", eeprom.get("version"))
    print("stereoEnableDistortionCorrection:", eeprom.get("stereoEnableDistortionCorrection"))
    print("stereoUseSpecTranslation:", eeprom.get("stereoUseSpecTranslation"))

Best,
Oskar

Seneral

Hey Oskar-Luxonis, thanks for the response! This is the output:

version: 6
stereoEnableDistortionCorrection: False
stereoUseSpecTranslation: True

Note that I do get a valid calibration (including distortion coefficients) for the color camera with each frame (from getTransformation). I explicitly tried that because I also suspected the camera to have lost the calibration values, but I don't think that's the case anymore.

Oskar-Luxonis

Seneral - thanks, that makes sense. The coefficients being present in getTransformation() is expected: they can still be available to the host, which is why your OpenCV path works. The issue seems to be that the on-device Camera-node undistortion path is gated by the EEPROM version/flags.

Since your EEPROM is version: 6, I’d suggest doing a fresh manual calibration with the current calibration tooling, then checking whether the resulting flashed user calibration is upgraded to the newer format/flags.

Docs:
https://docs.luxonis.com/hardware/platform/depth/manual-calibration

After calibration, please run this again, the snippet that I sent before.
Ideally we want to see version: 7, stereoEnableDistortionCorrection: True, and stereoUseSpecTranslation: False.

Seneral

Oskar-Luxonis Oh ok that does make sense, thanks. I am encountering a problem already described here:
luxonis/depthai1245

Traceback (most recent call last):
  File "/mnt/temp/BUILD/depthai/./calibrate.py", line 1255, in <module>
    Main().run()
    ~~~~~~~~~~^^
  File "/mnt/temp/BUILD/depthai/./calibrate.py", line 1246, in run
    self.capture_images_sync()
    ~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/mnt/temp/BUILD/depthai/./calibrate.py", line 920, in capture_images_sync
    combinedImage = np.hstack((combinedImage, subImage))
  File "/home/louis/.venv/depthai-update/lib/python3.14/site-packages/numpy/core/shape_base.py", line 359, in hstack
    return _nx.concatenate(arrs, 1, dtype=dtype, casting=casting)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: all the input array dimensions except for the concatenation axis must match exactly, but along dimension 0, the array at index 0 has size 239 and the array at index 1 has size 240

Namely, that comment suggests to use a patched script for OAK-D Lite:
luxonis/depthai1246c217e29?diff=split&w=0
I've given it a try, and it did allow me to capture all image, and the processing seemed to go well, too, but I had another concatenation issue just like that at the end.
I'll post details in that issue, if you don't mind, since it clearly seems to be related.

Beyond that, is there another way to update the EEPROM with an updated format using the factory calibration?
During capture, I was not able to move far away from the screen (as the page instructed), as the grayscale cameras were not able to detect enough squares past 80cm on my 24inch 1080p screen. So I'm not too confident I'll be able to get a good calibration anyway.
Perhaps if the parameters changed, I can try to manually fit one distortion model to another? Then we'd be able to make a straight EEPROM version update script with the same factory calibration.
Any help in that direction is appreciated.

Seneral

I have verified with the calibration_dump script that I do not have a factory calibration on my device, it appears as a user calibration.
Unfortunately the calibrate script never created a backup, so I lost my original factory calibration entirely. Yikes. I patched the script to create a backup.

With that terrible news out of the way, with the script patched to actually write the version number and two boolean flags correctly, it the on-device undistortion does seem to work again.
And my own calibration, despite having severe issues following the protocol, does seem to be good. Will do further tests.
I do have a backup of just the rgb cameras distortion coefficients and extrinsics, and they are the same format as the actual EEPROM, so I can revert just those if I feel the need to. Still, not great.

Thanks for your help. Would be really cool if the on-device undistortion was just transparently able to handle an older EEPROM version, or if there was at least a tool to auto-upgrade the version without changing the calibration. That's what versioning is for, after all.

Oskar-Luxonis

Hey Seneral - thanks for the detailed follow-up, and sorry about the calibration backup issue. Is this some very old device? It should have factory calibration.

Your result confirms the root cause: once the calibration was written as version: 7 with:

stereoEnableDistortionCorrection=True
stereoUseSpecTranslation=False

the Camera-node undistortion path started working again.

You shouldn’t need to fit one distortion model to another. If the original intrinsics/distortion/extrinsics are valid, a metadata-only migration should be possible in principle: keep the same calibration values and update only the EEPROM version/flags. The safe flow would be:

dump/backup the current calibration JSON
change only version, stereoEnableDistortionCorrection, and stereoUseSpecTranslation
test it first with device.setCalibration() so it is runtime-only
only then flash user calibration if the undistorted stream works

I’ll raise this internally. Thanks;
Oskar

Seneral

Oskar-Luxonis Yes, it's a kickstarter OAK-D Lite model. Sorry I didn't mention that.
If the version difference is so superficial, why is it a version difference in the first place? I figured this would be a change in distortion model altogether - something that does require user action.
IMO this should not warrant it silently failing if the only difference is one simple deterministic change of behaviour. At least that's from the limited information I have.
Anyway, thanks for the help, luckily the calibration did seem to go well.

Oskar-Luxonis

Seneral That makes sense, yeah. Those early OAKs had older calibration storage, that’s also why there isn’t a separate factory calibration to fall back to.
On newer devices, factory calibration is stored separately and shouldn’t be deletable in the same way, and newer calibration formats shouldn’t hit this issue.

Glad the recalibration worked in your case.

You can also check out our newest cameras. We’ve come a long way since then, especially with OAK 4.

Best,
Oskar