• RAE

  • No chance in getting a WLAN connection & device gets VERY hot

Hey, I am currently in progress of testing and trying to reproduce the issue - I am not really managing to get rae to go over 57 degrees even after couple of hours of running cameras+teleopt stack.

In the past we had issues with LEDs overheating the device - that issue should be solved but I think it is still worth a try to check if it is LEDs overheating the device and then go from there. You can turn LEDs off in default ros stack (robot.launch.py) by either removing LED peripheral node from rae_hw/launch/peripherals.launch.py (assuming you disabled the agent) or even bit bootleg solution like changing this line to always be false (thus sending empty LED messages) should suffice. If we can narrow down issue to since peripheral that would be very helpful.

Thanks and sorry for the inconvenience.

    DaniloPejovic, running a full ROS stack for about 35 minutes with a disabled LED node. WiFi temperature holds at ~58-59 degrees. Battery gauge - 54-55. Also, I gave a relatively small load on servos via teleop. So, your observation regarding LEDs seems correct, and they cause overheating. So what was the fix? Is it a pure hardware issue? And if it was fixed, then how did it appear in production? Anyway, what would be the next steps?

      As there were recent LED updates pushed to the ROS repo, I decided to check the theory and executed a full stack with LED node, which led me to the following numbers in just 5 minutes:

      However, there was another observation. In the previous message, I didn't use cameras. And when I added a couple of camera views in RViz, there was a temperature spike in the WiFi thermal zone.

      I could reach 66-67 degrees. However, it never jumped above this point. So, it seems like the problem is more complicated. LEDs are still probably the main failure point. But I don't believe they cause overheating in isolation. When I shut down the ROS stack, LEDs remained active (bug). But the temperature dropped to 57-58 degrees as well. So, it seems like cameras + LEADs in conjunction cause the overheating.

      Update: after a couple of hours of running the full stack with active cameras but w/o LEDs, I still reached the high temp in a WiFi zone (70-71 degrees). So it seems like it's just a matter of time to come to the red zone with active camera streams.

        Mike based on the specs, WiFi module operating temperature is 0-80 degrees. While processors start throttling when overheated, units like WiFi usually just shutdown until temperature is stabilized to protect themselves from the thermal shock (based on feedback from people who do a lot of stuff with such hardware).

        I didn’t find it in specs, but if I was the manufacturer, I’d probably leave ~5% buffer for thermal protection. It’s 76 degrees in this case (and it was +- the last value I saw before RAE rebooted). That’s why I’d treat ~72 degrees as a red zone for the end-user to get prepared for shutdown.

        From what I’ve seen on disassembled RAE photos, WiFi module is located in the zone close to one of the camera processors. Based on VPU specs, its operational temperature is up to 120 degrees. So 5 active processors and LEDs could generate a lot of heat for such a small area.

        • Mike replied to this.

            sskorol … Too early for those conclusions … An operating range is an operating range … if a device caused a reboot within the operating temp range then the device is faulty - it does happen.

                Mike I wonder if you've already tried to measure the temperature on your robot? What are the numbers in active mode with cameras, leds, teleoperation?

                • Mike replied to this.

                    sskorol … I have used the default app to drive Rae around until battery was flat … LEDs were on … WIFI on … did not measure temps … Rae worked fine … After seeing how raw things are at the moment and my available time I will shelve Rae for a month and dust off in December when I will have more time. Hopefully by then there is more software integration and updates … Meta Quest 3 integration soon ;-)

                      Hey, @sskorol while wifi operating range is up to 80 degrees and temperatures you are experiencing fall under it (so it shouldnt cause any issue while running the device) I havent been able to replicate that issue on few samples I have tried.

                      Are you experiencing any performance issues when RAE heats up?

                        DaniloPejovic well, RAE just reboots when I reach these numbers, so…

                        Also note that ~74-75 degrees is the last number I saw from the CLI tool for temperature measurement. Theoretically, it could have rapidly jumped higher but has never been reported due to the actual reboot.

                        If no one can reproduce it, I guess it's just a faulty device, and I may request a replacement, right?

                            Hi sskorol
                            Of course. Please write an email to support@luxonis.com, link to this thread and ask for a replacement. We will likely run some tests on the device, trying to figure out the problem.

                            Thanks and sorry for the inconvenience.

                            Regards,
                            Jaka

                              18 days later

                              Sorry for not coming back here for a time, but as stated above, I needed to exchange my rae first and got an unplanned two weeks business trip on top… My new device arrived today, will hopefully find time to test it tomorrow. Getting it hooked up to RobotHub and driving it around would already be an achievement over the last time 😉

                              Getting the device exchanged was no problem and the Luxions staff was really helpful, took a while, though…

                              I did give it a quick try: Good news is, that the display isn't flickering so far 😉 I tried setup via RH, QR code was recognized, trying to connect to my WLAN… Connection error. Tried again - connection error. Tried another WLAN/SSID - connection error. Followed the instructions to shut down (double press of the power button) and try again. This time the display shows "Connecting…", then the rae flashes red, then turns white and the display is stuck at "Connecting…" (with the "pulsing" dots. Shutdown with double press doesn't work anyomore either, only hart shutdown (8s press) works.

                              This is really no fun. Any idea why the rae is not connecting to my WLANs (passwords are from a password manager, so we can exclude mistyping)? Is the setup via RH even working for anyone?

                              Suggestions on how to continue?

                                Hi DiMa
                                I always set up the device using RH QR code, never had problems. I'd try factory reset if you are not getting anywhere with the setup. Perhaps there was an error somewhere in the process, which should get fixed by a reset. Is the WIFI password/SSID using any weird characters/white spaces/etc (should work anyway, just checking)?

                                Thanks,
                                Jaka

                                  No, just the usual mix of numbers, characters, special characters ({[]}…). No whitespaces, no UTF stunts. SSID is hidden, but a) that shouldn't matter and b) it doesn't work with non-hidden SSIDs either.

                                  Did a factory reset, I think - no change. BTW, how do I know, that the factory reset did work? I press the button 10s - and then? Nothing on the display, device doen't turn off…?!

                                  Some more troubleshooting via direct connection (USB-C):

                                  • After each reboot, hostapd is crashed.
                                  • Information in wpa_supplicant.conf is correct (SSID, password - no weird character translations as I read elsewehere etc. - everything is fine, that part of reading the QR code does definetly work)
                                  • No IP address obtained, not connected to my WLAN - in line with the "connection error" messages on the deveice, when I try the QR-code method.

                                  How to get it connected to the WLAN?

                                  systemctl stop hostapd

                                  systemctl restart systemd-networkd

                                  wpa_supplicant -B -i wlp1s0 -c /etc/wpa_supplicant.conf

                                  And then the rae is connected to my WLAN, can ping Google etc.

                                  Does this give any indication, what's wrong with the device/setup? The issue (as mentioned earlier in this thread!) seems to be a crashed hostapd. I'd like to disable it completely to check, if then maybe the connection comes up properly. Maybe then the QR-method would even work…? The display still states "Register at…

                                  I tried using the QR code method with a the WLAN connection up (via shell/USB-C) - same result (QR code is read, SSID/pwd is transferred, connection error… while the rae is connected to WLAN and can ping e.g. google.com). Provisioning app is running, too, I checked that.

                                  Very strange, I really don't get it 🙄

                                  Found this in the robothub-ctl logs:

                                  200 14:59:04 I user/agent/hub/config : calling home (expectedTeamId = null)

                                  200 14:59:04 I user/agent/hub/config : calling home -> first time, need to authorize

                                  200 14:59:05 W user/agent/hub-connection : callHome failed!

                                  {

                                  "error": {

                                  "cause": {

                                  "code": "CERT_NOT_YET_VALID",

                                  "message": "certificate is not yet valid",

                                  "name": "Error",

                                  "stack": "Error: certificate is not yet valid\n at TLSSocket.onConnectSecure (node:_tls_wrap:1540:34)\n at TLSSocket.emit (node:events:513:28)\n at TLSSocket.emit (node:domain:489:12)\n at TLSSocket._finishInit (node:_tls_wrap:959:8)\n at ssl.onhandshakedone (node:_tls_wrap:743:12)"

                                  },

                                  "message": "failed to obtain auth code",

                                  "name": "Error",

                                  "stack": "Error: failed to obtain auth code\n at Object.callHome (file:///usr/libexec/robothub/robothub-agent.mjs:219661:17)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async ensureCallingHome (file:///usr/libexec/robothub/robothub-agent.mjs:187910:28)\n at async Object.ensureCallingHome (file:///usr/libexec/robothub/robothub-agent.mjs:186087:9)\n at async Promise.all (index 0)\n at async file:///usr/libexec/robothub/robothub-agent.mjs:238084:9"

                                  }

                                  }

                                  200 14:59:05 I user/agent/hub-connection : waiting 69829 milliseconds

                                  Sounds to me like the device does try to register with the WLAN link up…?!

                                  And another test: When I

                                  • Mask hostapd (so it doesn't get started)
                                  • Log-in after reboot and restart systemd-networkd
                                  • Use the QR-method

                                  I get connected to my WLAN (so even the wpa_supplicant command is apparently executed correctly in the background) - but still "Connection error" on the display and nothing in RobotHub.

                                  I give up, need some pointers. This all makes no sense at all to me anymore by now.

                                    Hi DiMa
                                    This looks like the device's system clock is incorrect, leading it to believe that the certificate is not yet valid.
                                    Re-setting the internal clock should fix the error.

                                    Thanks,
                                    Jaka