RGB-Thermal Align DepthAI v3

cnorton

Hi,
I'm working on getting my RGB and Thermal images aligned. I've followed the example here for DepthAI v2 and converted the code to v3, but I still can't get it to work. I've also tried following the ImageAlign example here, but I can't get that to work either. Is there an example of RGB-Thermal Alignment for DepthAI v3. I can kind of getting it working host-side, but it's incredibly slow and would rather run it on the camera's processor.

Thanks!

OskarSonc

Hi @cnorton — quick question to narrow this down: were you able to get the original v2 RGB-Thermal Align example working as-is? And in v3, do you see an error or does it run but give mis-aligned frames?

Thanks
Oskar

cnorton

OskarSonc Hi, I just got it working with depthai 2.30.0. I ran the calibration step, and there's no warning now but the frames are slightly mis-aligned. I would like to get this working with 3.0.0-rc4 since I'm running my Oak-D 4 Pro as well in my setup, which is why I'm asking for an example.

I'm trying to start with the v2 script and port it to v3. I've followed the porting guide but I still seem to get dumb errors like:
AttributeError: 'depthai.node.Camera' object has no attribute 'setFps'

I'm sure the new version is much better than v2 but it's really confusing when trying to work with both versions.

OskarSonc

Hi @cnorton — we don’t have an official RGB-Thermal align example for v3 yet. In the meantime, you can lean on two resources:

v2→v3 porting guide (helps translate the old script):
https://docs.luxonis.com/software-v3/depthai/tutorials/v2-vs-v3/
Depth Align example (shows how to use the ImageAlign node in v3, which is essentially the same building block you’d use for RGB<->Thermal):
https://docs.luxonis.com/software-v3/depthai/examples/image_align/depth_align/

harris66

Hi, I have a similar issue here.

I’m trying to implement RGB–thermal alignment in DepthAI v3 using an RVC2-based thermal camera and my goal is able to tune staticDepthPlane suing runtime. The thermal stream works completely fine when I use the normal node::Thermal → Sync approach, and I can tune all ambient parameters (distance, emissivity, etc.). However, this Sync-only setup does not allow updating the staticDepthPlane at runtime, since that requires going through ImageAlign.

As soon as I switch to the v3 alignment pipeline:

thermal->temperature.link(align->input);

rgbOut->link(align->inputAlignTo);

align->outputAligned.link(sync->inputs["Raw_Aligned"]);

the aligned thermal output becomes all NaN, and min/max prints as -nan / -nan. No valid thermal pixels are produced. Eventually the device errors out with: Device already closed or disconnected: io error

I also test with readCalibration of thermalsocket, the values are valid so calibration is not the issue.

So my question: Is ImageAlign supported for RVC2 thermal input in v3? Also, is thermal-to-RGB alignment with tunable staticDepthPlane possible in hardware on RVC2, or must it be done entirely on the host?

OskarSonc

Hey @harris66 we now have DepthAi v3 thermal align example, you can check it out:
luxonis/depthai-coreblob/main/examples/python/RVC2/Thermal/thermal_align.py

Thanks,
Oskar

harris66

OskarSonc Do you have a c++ version of the thermal align example in DepthAI v3 using image align node?