Hi there,

My team has been working on various projects with OAK 1s and OAK 1 Maxs with success for a long time; however, unfortunately, there are some X_LINK errors that we consistently encounter and most of our efforts to mitigate these issues have proved ineffective. Some of the specific issues we are currently running into are detailed below:

1.      Programs will hang on the line where it tries to get a frame from the camera. No exception is thrown/nothing is printed, the program just stays stuck. Program has to be rerun. Can’t tell what the error is.

2.      Immediately upon trying to start up the camera, the error “failed to find device after booting, error message: X_LINK_DEVICE_NOT_FOUND” is thrown. Upon checking lsusb, can tell that the camera is still plugged in. Must be solved through unplugging and replugging in the camera.

3.      Will receive “failed to find device, error message: X_LINK_DEVICE_NOT_FOUND”, not necessarily with “after booting”, after a longer period of time like a week. Thought this was solved through periodically restarting the program but unfortunately, camera still has to be unplugged and replugged in.

4.      Program crashes seemingly randomly, sometimes after a few days, a few hours, or a few minutes. The error occurs when trying to get a frame from the camera and says “RuntimeError: Communication exception – possible device error/misconfiguration. Original message ‘Couldn’t read data from stream: ‘rgb’ (X_LINK_ERROR)”. Trying to then close the device in order to recreate it and restart the pipeline leads to “failed to find device after booting”. On lsusb, can see the camera remains plugged in. Restarting the program doesn’t necessarily help, but strangely enough, after leaving the camera alone for a length of time and then retrying with changing nothing else, it works.

 

Cameras have been on 5m USB extenders and longer, externally powered ones. Our environment experiences some vibration and dust. We are using DepthAI 2.29 in Python. Camera pipelines are being made in both main threads and others. Rebooting computing unit doesn’t help.

Any information at all around the cause of these issues or suggestions for solutions would be hugely appreciated. Unsure if it is a hardware change we have to make or if there is a software solution to avoid having to unplug and replug in the camera.

Thank you.

 

    hatcob Programs will hang on the line where it tries to get a frame from the camera. No exception is thrown/nothing is printed, the program just stays stuck. Program has to be rerun. Can’t tell what the error is.

    and

    hatcob Program crashes seemingly randomly, sometimes after a few days, a few hours, or a few minutes. The error occurs when trying to get a frame from the camera and says “RuntimeError: Communication exception – possible device error/misconfiguration. Original message ‘Couldn’t read data from stream: ‘rgb’ (X_LINK_ERROR)”. Trying to then close the device in order to recreate it and restart the pipeline leads to “failed to find device after booting”. On lsusb, can see the camera remains plugged in. Restarting the program doesn’t necessarily help, but strangely enough, after leaving the camera alone for a length of time and then retrying with changing nothing else, it works.

    Are knows stability issues, we are working on a fix / better logging for them.

    hatcob 2.      Immediately upon trying to start up the camera, the error “failed to find device after booting, error message: X_LINK_DEVICE_NOT_FOUND” is thrown. Upon checking lsusb, can tell that the camera is still plugged in. Must be solved through unplugging and replugging in the camera.

    3.      Will receive “failed to find device, error message: X_LINK_DEVICE_NOT_FOUND”, not necessarily with “after booting”, after a longer period of time like a week. Thought this was solved through periodically restarting the program but unfortunately, camera still has to be unplugged and replugged in.

    These two indicate a power issue. When device is unbooted, it is in rest mode which consumes very little power. When booting it with an app, the device reboots and reconnects as a different product ID usb device, which also consumes much more power. When not enough (or slighly not enough) power is supplied, the device browns out.

    Thanks,
    Jaka

            Hi Jaka,

            Thank you for the quick reply.

            jakaskerl hatcob Programs will hang on the line where it tries to get a frame from the camera. No exception is thrown/nothing is printed, the program just stays stuck. Program has to be rerun. Can’t tell what the error is.

            This, fortunately, can be resolved through restarting the program.

            jakaskerl hatcob Program crashes seemingly randomly, sometimes after a few days, a few hours, or a few minutes. The error occurs when trying to get a frame from the camera and says “RuntimeError: Communication exception – possible device error/misconfiguration. Original message ‘Couldn’t read data from stream: ‘rgb’ (X_LINK_ERROR)”. Trying to then close the device in order to recreate it and restart the pipeline leads to “failed to find device after booting”. On lsusb, can see the camera remains plugged in. Restarting the program doesn’t necessarily help, but strangely enough, after leaving the camera alone for a length of time and then retrying with changing nothing else, it works.

            This one, however, is more urgent. Do you have any suggestions for the time being from what you know about the error for either how we can try to prevent it occurring or what can be done after it is caught? What leads to this error? Does the last part about it working again after waiting make any sense? Would appreciate any more information you have currently about the known stability issue.

            jakaskerl These two indicate a power issue. When device is unbooted, it is in rest mode which consumes very little power. When booting it with an app, the device reboots and reconnects as a different product ID usb device, which also consumes much more power. When not enough (or slighly not enough) power is supplied, the device browns out.

            For these issues, why does this lead to needing to unplug and replug the camera and not just stopping the program (unbooting) and restarting it when the camera is still being supplied power? Is there a way to measure/return the amount of power being supplied to the device?

            Thank you for your help.

                    hatcob Do you have any suggestions for the time being from what you know about the error for either how we can try to prevent it occurring or what can be done after it is caught? What leads to this error? Does the last part about it working again after waiting make any sense? Would appreciate any more information you have currently about the known stability issue.

                    Difficult to say, but it seems like there is a communication issue at certain high temperatures. Could also be that the power supply you are using is not capable enough to maintain power level when device resistance increases due to temperature. This would make sense when looking at “failed to find device after booting" issue as well. How do you power your devices? Perhaps a slightly cooler environment or better cooling might help - would be interesting to debug.

                    hatcob For these issues, why does this lead to needing to unplug and replug the camera and not just stopping the program (unbooting) and restarting it when the camera is still being supplied power? Is there a way to measure/return the amount of power being supplied to the device?

                    USB meter. No way to do that from the device.

                    Thanks,
                    Jaka

                          jakaskerl

                          Hi Jaka, thank you for your response.

                          jakaskerl Perhaps a slightly cooler environment or better cooling might help - would be interesting to debug.

                          I see that the ambient operating temperature of the OAK 1 is up to 50degC and the chip can tolerate temperatures up to 105degC. I assume the function device.getChipTemperature().average returns the average temperature of the RVC2 chip. Is there something better to monitor than average? Currently it seems to be stable around 54degC which is quite far off from 105. Could this still be the issue?

                          jakaskerl How do you power your devices?

                          One project's camera experiencing this issue is powered off a 5m USB3.0 extender coming directly from the computing unit. How can we ensure it is keeping power consistent with possibly changing temperatures?

                          Do you have any suggestions for cooling the OAK 1?

                          Thanks again.

                                  hatcob
                                  Temperatures of the chip are fine and expected. I don't think it's failing due to high temps, but because of the additional resistance in circuits of the device - this was my (perhaps far fetched idea). If the power was limited, this could cause the issue and why it would be a problem when device was immediately rebooted again (still hot).

                                  hatcob One project's camera experiencing this issue is powered off a 5m USB3.0 extender coming directly from the computing unit. How can we ensure it is keeping power consistent with possibly changing temperatures?

                                  Your best bet is to check the power at the device with a USB power meter. Perhaps the extender isn't able to supply power..

                                  Thanks,
                                  Jaka

                                      10 days later
                                      5 days later

                                      anze_jakos

                                      Hi @anze_jakos , thank you as well! Not all of our OAKs seem to have that screw hole. Is that a newer addition?

                                      Thank you.

                                        Hi @hatcob we have introduced screw in locking mechanism on our OAK1 lineup to enchance resistance againts high vibration enviroments already a few years ago and I would highly recommend using whenever is possible. Also on OAK-D.