Depth estimation inaccurate (occasionally)

MrX

Hey guys, I'm back 😉

(or rather I saw the announcement, that I should write here on the forums now instead on discord)

This one is again a bit difficult to describe/put in in words, but I'll try anyway.

We're basically in the stage to make the issue at least reproducible.

So my colleague has noticed some odd behaviour:

Usually the cams work all fine with the depth estimation,

but in some cases, the estimation is way off (like 30% or so).

Difficult to have stats right now, he can only describe it currently by the Anchorman meme "60% of the time, it works every time"

It seems after starting it's either working fine all the time,

or works "ok" all the time,

or the measurements are way off all the time.

So we suspect some calibration issue after starting the script.

(I will try to get it and post it here too, when I have it)

current infos that I got so far:

- dai library versions are the latest

- happens on win & linux

- tested on 2 different cams (oak-d-s2)

So it'd be interesting, if anyone else encoutered anything similar (I haven't seen any recent posts with anything like this)

and would be nice, if you guys could provide some suggestions/ideas how we should debug this issue/ what debug data to collect, etc ...

(I'm thinking of some setup with a fixed position of the cam,

we trigger the script, as long the output fits the expected values, we just shutdown and restart it,

and in case the measurements are off, we collect debug data)

jakaskerl

Hi MrX
What changes for depth estimation to be wrong - is there something in the environment (perhaps a homogeneous surface)? Does the issue persist while the device is alive, or does it suddenly appear, seemingly out of nowhere? Since you have said "all the time" i will assume the accuracy remains the same.
This means the issue is likely caused at boot-up which would point toward the calibration script issue.

Can you check the internal device memory for different calibration configurations? Perhaps there are different calibrations being used at boot.

Try to use https://docs.luxonis.com/projects/api/en/latest/samples/calibration/calibration_reader/?highlight=eeprom#calibration-reader to look at calibration config and look for a change when the error occurs.

MrX tested on 2 different cams (oak-d-s2)

^ leads me away from calibration unless cameras were calibrated more than once - factory and user calibrations

It could also be a FW issue, but let's check other things first.

Thanks,
Jaka

MrX

Hi, thx for the fast reply last week.

I just wanted to give a quick update. My colleague was quite busy with a lot of things this week, but the calibration reader example was helpful. He did some tests on the side and something with the configuration is off indeed, when the error occurs.

jakaskerl Does the issue persist while the device is alive, or does it suddenly appear, seemingly out of nowhere? Since you have said "all the time" i will assume the accuracy remains the same.
This means the issue is likely caused at boot-up which would point toward the calibration script issue.

Regarding that, yes, I was not sure how to word it precisely, that's why i said "all the time", meaning indeed, it does not suddenly appear/change, but it is persistent while the device is alive. So the accuracy remains the same for the whole run of the script (and is either working fine, "ok", or way off)

My colleague is currently working on a MRE that we can post here and I'll soon post what the exact output/configuration is, when the error occurs.

MrX

Hello everyone, we have some updates:

So, I was not able to replicate the issue with the 3 cameras that I have.

But my colleague noticed, when started cold (or cooled down), there is usually no issue, but when the cameras were running (and actually extremely hot), and then restarting, the issue would occur.

So we just used the calibration reader script in the end / I just added some temp readings, too.

Here is one of the problematic example output:

[[0.0, 0.0, 0.0], [0.0, 0.0, 0.0], [0.0, 0.0, 1.0]]

3840

2160

RGB Camera Default intrinsics...

[[3093.198486328125, 0.0, 0.0], [0.0, 0.0, 0.0], [0.0, 0.0, 1.0]]

3840

2160

RGB Camera resized intrinsics... 3840 x 2160

[[0.00000000e+00 0.00000000e+00 0.00000000e+00]

[0.00000000e+00 0.00000000e+00 0.00000000e+00]

[0.00000000e+00 0.00000000e+00 1.00000000e+00]]

RGB Camera resized intrinsics... 4056 x 3040

[[3.26719092e+03 0.00000000e+00 2.03175232e+03]

[0.00000000e+00 3.26719092e+03 1.51714795e+03]

[0.00000000e+00 0.00000000e+00 1.00000000e+00]]

LEFT Camera Default intrinsics...

[[796.7051391601562, 0.0, 645.4278564453125], [0.0, 796.7051391601562, 408.0694885253906], [0.0, 0.0, 1.0]]

1280

800

LEFT Camera resized intrinsics... 1280 x 720

[[796.70513916 0. 645.42785645]

[ 0. 796.70513916 368.06948853]

[ 0. 0. 1. ]]

RIGHT Camera resized intrinsics... 1280 x 720

[[799.68243408 0. 622.44793701]

[ 0. 799.68243408 352.41833496]

[ 0. 0. 1. ]]

LEFT Distortion Coefficients...

k1: 0.0

k2: 0.0

p1: 0.0

p2: 0.0

k3: 0.0

k4: 0.0

k5: 0.0

k6: 0.0

s1: 0.0

s2: 0.0

s3: 0.0

s4: 0.0

τx: 0.0

τy: 0.0

RIGHT Distortion Coefficients...

k1: 0.0

k2: 0.0

p1: 0.0

p2: 0.0

k3: 0.0

k4: 0.0

k5: 0.0

k6: 0.0

s1: 0.0

s2: 0.0

s3: 0.0

s4: 0.0

τx: 0.0

τy: 0.0

RGB FOV 68.7938003540039, Mono FOV 71.86000061035156

LEFT Camera stereo rectification matrix...

[[ 0.00000000e+00 0.00000000e+00 -2.76791411e+01]

[ 8.31704614e-03 0.00000000e+00 -2.67830176e+01]

[ 3.49135028e-06 0.00000000e+00 9.95629622e-01]]

RIGHT Camera stereo rectification matrix...

[[ 1.00213630e+00 0.00000000e+00 -2.29311345e+00]

[ 8.28608097e-03 0.00000000e+00 -9.51243515e+00]

[ 3.47835164e-06 0.00000000e+00 9.95814852e-01]]

Transformation matrix of where left Camera is W.R.T right Camera's optical center

[[ 9.99932587e-01 0.00000000e+00 -9.42897983e-03 -7.48859406e+00]

[ 6.68985024e-03 0.00000000e+00 -9.12995916e-03 2.67834123e-03]

[ 9.49023943e-03 0.00000000e+00 9.99913871e-01 -5.00004739e-02]

[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 1.00000000e+00]]

Transformation matrix of where left Camera is W.R.T RGB Camera's optical center

[[ 9.99960124e-01 -8.50090384e-03 2.72128452e-03 -3.77367687e+00]

[ 0.00000000e+00 9.99903500e-01 1.10077411e-02 1.00632086e-02]

[ 0.00000000e+00 -1.09842541e-02 9.99935687e-01 -1.20922260e-01]

[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 1.00000000e+00]]

The temperature thing reminded me of a post I did read last year luxonis/depthai-python522

He disassembled both cameras, and the thermal pads were missing in both cameras. As pads were not available in shops, we just quickly tested with thermal paste and it seems the problem disappeared (for now…)

Meanwhile, all 3 of my cameras have thermal pads, that's why they usually peak at only 70° , though some metal cuttings from the machining fell out when I opened the devices …

jakaskerl

Hi MrX
I fail to see how temperature would have an effect on depth perception. Usually a device will stretch and skew a bit when heating up, and also the lens may get some additional distortion but the overall effects would (i presume) be minimal.
Exactly how off are your measurements?

Thanks,
Jaka

MrX

Hi,

well, that's what we have found so far.

The measurements were off ~30% or more and I don't think the thermal expansion of the device is the problem here.

Our suspicion is, that due to overheating issues, something went wrong when reading the parameters. As you can see the Distortion Coefficients are all 0 in that output, and that causes the various matrices to be 0 or mostly 0. I didn't have that problem with my 3 cameras that had the thermal pads and thus couldn't reproduce the issue either.

erik

Hi MrX ,
This does indeed look like a bad calibration. What's the MxID of the device (script here)? This would allow us to check internal calibration logs. I would suggest recalibrating the device.
Thanks, Erik