ToF sensor Known Working Test Script Fails on my device

Nearpoint

Hello,

I am trying to utilize the ToF sensor with my OAK-FFC-3P board which I purchased last year.

I have depthAI updated to the latest version including all the dependencies but for some odd reason when I try to run the test tof script available in the documentation I get the following error:

(depthai_env2) feroze@feroze-aven-linux:~/depthai-python/examples/Camera$ python3 tof_test.py
2.27.0.0.dev0+21fe58ab7c9c640e5caa77887348b4dd5a98e441
Connected cameras: [{socket: CAM_A, sensorName: IMX577, width: 4056, height: 3040, orientation: AUTO, supportedTypes: [COLOR], hasAutofocus: 0, hasAutofocusIC: 1, name: color}, {socket: CAM_B, sensorName: LCM48, width: 5312, height: 6000, orientation: AUTO, supportedTypes: [MONO], hasAutofocus: 0, hasAutofocusIC: 1, name: left}, {socket: CAM_C, sensorName: S5K33D, width: 640, height: 480, orientation: AUTO, supportedTypes: [MONO], hasAutofocus: 0, hasAutofocusIC: 0, name: right}]
[18443010F1FEF30800] [1.2.2.4] [3.483] [ToF(0)] [error] ERROR unexpected frequency code: 255

[18443010F1FEF30800] [1.2.2.4] [3.483] [ToF(0)] [error] ERROR unexpected phase code: 255

[18443010F1FEF30800] [1.2.2.4] [3.483] [ToF(0)] [error] Orientation not supported! Only normal and 180 flip(not mirrored) can be used.
[18443010F1FEF30800] [1.2.2.4] [3.490] [ToF(0)] [error] ERROR unexpected frequency code: 105

[18443010F1FEF30800] [1.2.2.4] [3.490] [ToF(0)] [error] ERROR unexpected phase code: 21

[18443010F1FEF30800] [1.2.2.4] [3.490] [ToF(0)] [error] Orientation not supported! Only normal and 180 flip(not mirrored) can be used.
[18443010F1FEF30800] [1.2.2.4] [3.492] [ToF(0)] [error] Couldn't read CCM EEPROM: I/O error
[18443010F1FEF30800] [1.2.2.4] [3.494] [ToF(0)] [error] Couldn't read CCM EEPROM: I/O error
[18443010F1FEF30800] [1.2.2.4] [3.494] [ToF(0)] [error] Error: fppn_80 Could not read header data.
[18443010F1FEF30800] [1.2.2.4] [6.357] [ToF(0)] [error] ERROR unexpected frequency code: 211

[18443010F1FEF30800] [1.2.2.4] [6.357] [ToF(0)] [error] ERROR unexpected phase code: 220

[18443010F1FEF30800] [1.2.2.4] [6.357] [ToF(0)] [error] Orientation not supported! Only normal and 180 flip(not mirrored) can be used.
[18443010F1FEF30800] [1.2.2.4] [6.357] [ToF(0)] [error] ERROR unexpected frequency code: 7

[18443010F1FEF30800] [1.2.2.4] [6.357] [ToF(0)] [error] ERROR unexpected phase code: 204

[18443010F1FEF30800] [1.2.2.4] [6.357] [ToF(0)] [error] Orientation not supported! Only normal and 180 flip(not mirrored) can be used.
[18443010F1FEF30800] [1.2.2.4] [6.380] [ToF(0)] [error] ERROR unexpected frequency code: 255

Do I have to update the OAK-FFC-3P board in any way? I could not find instructions on how to do that if necessary, please point me to the right steps. Otherwise what could be the issue, I could not be the code since I am using the direct test script from the website. The ToF sensor is connected to CAM_C connector.

The ToF sensor itself was purchased early last year as well not sure if there is firmware on the ToF sensor itself I need to update. Regardless I am pretty confident this is a hardware issue. Please let me know what can be done.

Thanks
Feroze

jakaskerl

Nearpoint

Could you post the MRE of the code you are using?
Try connecting it to CAM_A.

But the depth you will receive will be bad since the modules from before March 2024 have not been calibrated. Once you get it working, if you wish for accurate depth, I'd suggest you write to support@luxonis.com so we can replace the module with a flashed one.

Thanks,
Jaka

Nearpoint

jakaskerl

It might be related to this: https://discuss.luxonis.com/d/2356-tof-bad-accuracy/45

The ToF sensor does work when connected to CAM_A

I am wondering do you know and can you confirm that the ToF sensor will be compatible with the IMX577 and IMX378 or IMX582.

This is very important to us that is it compatible

Luxonis-Alex

@Nearpoint There was a bug causing this [error] Couldn't read CCM EEPROM: I/O error in certain camera permutations, when ToF was not on CAM_A, related to optional CCM EEPROM missing on some color sensors. A fix was pushed on develop branch luxonis/depthai-python7629453 , could you give it a try, depthai version currently:
2.28.0.0.dev0+76294530fd0bf91b09469490a1d14871c374630a

If you need to use IMX582 (LCM48), that one needs 4 lane, so CAM_A (or CAM_D on OAK-FFC-4P).

A setup for the sensors you need could look like:

CAM_A: IMX582 (or IMX378)
CAM_B: IMX577 (sometimes it may be identified as IMX380, that's fine)
CAM_C: S5K33D ToF

and can be tested with commands like:
python3 utilities/cam_test.py -cams camc,t --- ToF only
python3 utilities/cam_test.py -cams camb,c camc,t --- both sensors on the shared I2C bus (B and C ports)
python3 utilities/cam_test.py -cams cama,c camb,c camc,t -rs -ds 2 --- all

Nearpoint

Thank you so much that is very helpful! We are approaching this as the final design target. Would there be any benefit to utilizing the OAK-FFC-4P over the 3P if we are certain the final product won't have more than 3 cameras? The 3P makes sense, but originally I was suspecting I might have to go with the 4P due to these issues but the 3P should work fine?

I will test with the updated branch as soon as I can, and let you know how it goes, thanks!

Luxonis-Alex

I think that 3P should work, as long as the other camera paired on the shared-I2C-bus sockets (CAM_B/left and CAM_C/right) doesn't have an EEPROM that would interfere with the ToF's module EEPROM, usually they're all at the 7-bit I2C address 0x50. For example with IMX577 and ToF on those sockets, if the IMX577 module (Arducam?) has an EEPROM, that's a problem.

You can run for example with IMX577 only attached, and the environment variable DEPTHAI_DEBUG=1, there's a message if an EEPROM was not found, I can have a look at the logs.
(A note that if a camera is mounted on CAM_B, but CAM_C is empty, the same camera from CAM_B would appear as detected on CAM_C, that's a quirk from the probing and can be ignored. It's related to the same I2C bus and reset of CAM_B not being re-asserted after probing)

Nearpoint

Luxonis-Alex

We plan to purchase all our cameras directly from Luxonis. Will this strategy avoid the EEPROM issues you mentioned, given that all components will be designed to work together on your modules?

We are buying the IMX577 M12 sensor from Luxonis but intend to remove the M12 lens and its mount entirely since we're pairing the sensor with a C-mount lens.

I want to emphasize our requirement for Luxonis to provide sensors that are compatible with various lenses, including C-mount lenses, which necessitates that any M12 lens mount be easily removable. There's a trend towards integrating lenses with sensors, but I hope Luxonis will continue to offer sensor modules where both the lens and the mount can be detached. Currently, with the AR0234 module, while the lens itself is removable, the mount is glued in such a way that makes its removal impractical. We prefer C-mount lenses for their superior image quality, which is critical for our application. We're developing a custom autofocus mechanism and using a custom lens with the IMX577, hence our desire for Luxonis to always support the sale of sensors with detachable lenses and mounts.

We are integrating depth data from a Time-of-Flight (ToF) sensor to control the sensor-to-lens position in our custom autofocus system for the IMX577. This setup leverages Luxonis technology to overlay depth information onto the color video frame, enabling an adaptive autofocus feature that detects and tracks features, adjusting focus as magnification changes or as objects move. If a feature of interest is outside the current depth of field, selecting it will automatically refocus. We're particularly interested in whether the ToF sensor can detect depth differences as small as 2-3mm from a distance of 220mm from the circuit board, which is crucial for inspecting details like text on capacitors or taller resistors that might otherwise be out of focus with a large aperture.

Could you provide insights on the ToF camera's performance in this scenario? As we await our reflashed unit, I'm eager to hear your thoughts on the ToF sensor's effectiveness in tracking such minute depth differences.

We've been working hard to integrate this technology, and understanding our use case might help ensure our specific needs are met. The ability to use custom lenses and mounts, and even to design our own autofocus systems, highlights the importance of Luxonis offering sensors with removable mounts and lenses. Additionally, any insights you could share about the ToF depth sensor's accuracy in this context would be immensely valuable.

Nearpoint

Luxonis-Alex

A minimum Z distance of 200mm is currently acceptable for our application; however, we could enhance user benefits if the ToF (Time of Flight) camera could reliably function at a reduced minimum Z distance of 150mm.

Our objective is to produce a reasonably accurate topography of circuit boards, which feature through-hole components generally between 3 to 5 mm in height. While there might be challenges with very thin surface-mount components, the camera should still perform adequately with components at least 3 mm tall when operating from the current minimum Z of 200mm.

This performance expectation assumes optimal lighting conditions, utilizing OLED side lighting to reduce reflections and ensure a clear view of the object on a white background. However, if we were to lower the minimum Z to 150mm, would we still achieve reasonable accuracy, or does the accuracy degrade significantly below 200mm?

Nearpoint

Luxonis-Alex

Oh, one more thing, we also plan to synchronize the frames across all three cameras purchased directly from Luxonis. Although I understand that Arducam manufactures the IMX577, and I'm aware that the fsync lines do not connect to the OAK board. Is there any chance you might produce your own IMX577 sensors with integrated fsync lines? We believe it would be highly beneficial to synchronize all frames between the IMX577, the ToF 33D Camera, and either the IMX378 12MP autofocus camera or the IMX582 32MP autofocus camera. Being able to easily hardware-sync frames for all these cameras, all purchased directly from Luxonis, would significantly enhance our project.