RobotHub anomalies and rae firmware update

gregflurry

I finally got the time to work with my rae again. When I last used it, it was "hung" trying to install and uninstall apps. Today I charged it and was prepared to do a hard reset.

I started RobotHub to see if the thing was still "hung". To my great surprise, despite the fact that I have only 1 rae, RobotHub showed 2 devices, both named `'keembay'. The status of the 'top' device was Online. The status of the 'bottom' device was Outage, and it was last seen 5 months ago, which is the last time I tried to work with rae. The Outage device indeed showed some presence of the two apps I tried to install.

The Online device appeared to be OK, so I installed the default app. I was able to get it to run and see the output of both front and back cameras. Progress!

I had to run an errand and put my computer to sleep for 2 hours. I did not power down the rae. I restarted my computer and RobotHub showed 3 devices! The 'top' was Online, and other 2 were Outage. The last was seen 5 months ago, and the middle was seen 2 hours ago.

The Online rae had no apps, so I had to again install the default app. It took a couple of tries, but it too eventually worked.

So, 5 months ago, RobotHub behaved very differently, and has clearly been updated. I appreciate that its new behavior solved the "hung" problem. However, it seems wrong that it basically ignored what was a functional device and flushed the app that I had loaded to create a new environment; that seems counterproductive. Am I missing something? Also, I could find no way to remove the Outage devices, and as some point I'm likely to have many more with RobotHub's current behavior of creating a new device every time it opens/restarts/scans or whatever it does. Is all this expected behavior?

Also, all of the devices, Online or Outage, are at FW Version 1.12. I suspect that is not the latest. I tried searching this group for how to upgrade. I found a post that suggests the RobotHub will show an "upgrade button" but nothing else. I cannot find that button anywhere in the UI for the RobotHub I am running. What am I missing?

Thanks.

jakaskerl

cc @MartinMotycka Thanks!

MartinMotycka

Hi.

Since we're unsure of the current state of your RAE and it's behavior is very unusual, I would like to suggest beginning from the scratch. That means FW update, hard reset and new connection to the Luxonis Hub (there was a rebranding few days ago).

You are right. The latest FW version is 1.14. You can update FW manually without the Hub. It's quite ease, please follow these instructions.
After FW update please do a hard reset of RAE. This can be done by pressing a button at the bottom of RAE with a needle for 10 seconds.
Delete all "RAE devices" from your team. That can be done from the device overview through icon in top right corner:
Connect RAE to the Hub.

Please let us know if this helped.

Thanks,

gregflurry

MartinMotycka I mistakenly posted my attempt to execute your suggestions to the general thread, so I'm replying to you to make sure you see it. Thanks.

gregflurry

Here is a report at my attempt to do what you suggested. I don't know if this matters, but I am running on a MacBook Pro with Apple silicon, using macOS, and Safari as the browser.

I attempted to upgrade the OS. This was a bit tricky, as I had to do a bit of searching to find the user ID/PW to use to SSH into rae (root and no password). I executed the command shown in your link. Based on the feedback from the command execution, it was not obvious if it had failed or succeeded. I must add that I did this over wifi, because I could not determine how to do it with the USB cable attached to my MacBook. Is that a better way?

I subsequently opened the Hub and deleted the two outage devices. I peeked at the online device, and it indicated OS 1.14, so I guess that worked. I then deleted the online device from the Hub.

I then attempted to reset rae. I found it very difficult to press the reset button, and I could not detect any obvious action caused by reset. I then attempted to connect to the Hub again, but failed. I pinged the device at its original IP address and that succeeded, so I figured I actually did not reset rae.

I tried again to reset. Again, it was very difficult (bad eye/hand coordination?). While the LEDs on the bottom of the device never changed, I did eventually notice that the device display went blank and then eventually said rae. I pinged again and got nothing.

I again attempted to connect to the Hub. It took a bit, but then seemed to connect. I attempted to install the default app (2.0.0). The Hub said the download was happening, then eventually showed a warning about taking a long time and network speed. I could ping the device on its new IP address.

The warning also said something about restarting the download. I could find no way to do that in any UI in the Hub, so I simply turned off the device. Eventually the Hub showed the device status as Outage.

After a couple of hours, I turned on the device. Unlike what I remember seeing earlier, the lights did not ever come on. Meanwhile, I went to the Hub. More anomalous behavior. In the main area, a dialog box said "Upgrade in Progress...". It also said "Waiting for device to connect...". A progress bar indicated something was happening, for moving from 23-something to 24-something. Shortly after, I noticed the Hub indicating the device was online (but still no indication of that from the device). Then in the dialog, a sub-box said "Upgrade is taking longer than usual...". The progress bar reached what I'd interpret as done. I waited maybe 10 minutes and nothing changed. The Hub indicated online, but also waiting to connect; progress bar complete, but still upgrade in progress.

I refreshed the Hub, and got the upgrade dialog again. This time the Hub indicated the device was "Outage". I noticed this time the dialog "Upgrade is taking longer than usual..." popped up just as the progress bar reached done. Having lost my patience, I closed the browser window and shut down the device.

I turned on the device again; a blue light flashed when I pressed the power button. But after several minutes, all I see is rae in the display, no lights. I opened a new browser window and started the Hub. Nothing on the device changed and the Hub reported 1 device online. In the Hub devices, the rae showed up as online. I could also ping the device, but the response times were very long (I'm on a wifi network of at least 100 MB fed by a 1 GB internet connection). I clicked the device and saw an indication that an app install had failed.

I uninstalled the failed app; I found how to do that by accident. I then tried to install v1.2.1. The Hub indicated the app got downloaded, and then said the app was initializing -- and the LEDs came on! And the Hub then indicated the app was running. I could see video from both the front and back cameras! So far so good -- but it was very painful to get to this point! What the heck was all that "upgrade" stuff?

I stopped the app. The lights went out. I guess that is correct. I shutdown the device. I noticed that the Hub, even after 3-4 minutes still indicates the device is online, but finally did indicate Outage. I also noticed that in the device information, a column named Agent said "23.223.1855 / RAE". That looks like the number that was trying to get upgraded. So clearly my device is still not all the way up to date. Why was I finally able to get past the attempt to upgrade? Why did the upgrade fail?

I turned the device on again. I waited a few minutes for the Hub to see it and indicate Online. When it did come Online, the LEDs turned on. When I looked at the device details there was no attempt to upgrade the agent. Why not? Is there a way to force that? Should I do so?

I noticed the app was running (I guess the LEDs indicate that). I was able once again to look at video from both the front and back cameras. As I'm sure has been reported, there was a significant lag from real time, and a slow frame rate.

So I guess your suggestions have solved my immediate problem. I can now use some version of the default app somewhat reliably. Clearly my experience indicates there are still some bugs or coin-flips in the device and maybe Hub. My experience has raised a number of questions in the way-too-long dialog above. But there are many more:

why is everything so slow? could there be more status indication, either in the the Hub or on the device display?
what version of the default app should I be using? from above, 2.0.0 would not even download for me, at least on the first attempt
what of the prebuilt apps should work? how do I find out what they actually do?
can the device only have on app at a time? this is suggested by the fact that a device has to be assigned to an app at some point, tho the only time that may work is when the app gets installed
if multiple apps can be installed is there a proper way to reassign the device?
what is the proper shutdown procedure? should I first stop any running app? should I shutdown using the power button on the device? Or should I use the shutdown button I discovered in the Hub?
in the Finishing Up section, there is a mention of "direct control". what does that mean? what can I control?
when I enter the URL, I see a web page saying "Robot Hub Agent". is that the same agent that was trying to get upgraded? I see an icon for the Default app, and for "Login to manage". is the latter the same as direct control? I am a bit hesitant to click anything for fear of getting myself messed up again.

Thanks again for the help. I really do appreciate it. But I have to say, that for something that was supposed to be for everyone/anyone, I am not sure that has been achieved. But is does seem better now than 5 months ago, even with all the remaining mysteries.

MartinMotycka

gregflurry
1." because I could not determine how to do it with the USB cable attached to my MacBook. Is that a better way? " Yes, it is. Cable is more robust solution in such cases. you can use the side USB-C for that. Please check this part of docs.

2. Reset RAE
The resetting button is on bottom. There is some resistance, so you should be able to feel, that you pressed that.

3. Turn off the device
It should be on the same place as device "delete"

If you shutdown RAE this way / or via one short press of the power button you initialize a shutdown process where all services are properly ended. In this case it took about 2 min before the device was shown as outage. If you turn it off hard-way (long press of power button) it will take a bit longer to show the outage.

4. Installing apps / Upgrades
Apps are installed directly on the agent. In this case it is the RAE. Luxonis Hub just provides an API on how to do that remotely without direct access. It is some kind of OTA update when the device downloads updates/apps via WiFi it's connected to. Please consider that some apps can be quite big. Especially if it is necessary to download the docker container. It can be several gigabytes. So it takes some time to download and install. A little bit of patience is needed here 🙂

5. RAE Agent
For RAE is the agent part of the FW. So it can not be updated by itself. Version 23.x is the correct one.

6. The lag
It is quite usual that there is a lag due to data transfer. It depends how big the lag is? If it's 1-2 seconds in video stream, it is OK.

7. The Default app
The Default App is a demo app that shows all of the RAE's capabilities in a general way. You can control movement of the RAE, lights, turn on “follow me” mode etc. If you open the RAE's IP address in the browser, you open the agent. You can see apps that are installed, when you login, go to Apps you will have there possibility to open app interface:

8. Apps from the store
You can have more apps installed at time. But only one can be active at time. If you want to change it, you have to go to “perception apps -> reassign OAK” and remove it there. After that you can select different app.

If you have other question, please do not hesitate to ask.
Thanks, M.

Dominus

More interesting that about 6+ months there is still nothing that was promised is maded, and even MOST BASIC features like firmware update need to be made with help of voodoo. What to say… there is even manuals not working, and even if you dig and read that there is new links to them they is outdated / wrong / not working and with current product state just unusable. There is actually nothing that worked on rae and sad to say - nothing useful can be made with rae in current state.

Why not make basically normal guides, working robothub with worked example apps? Not finish chatgpt, slam? Make even 1 functional that can be useful and not been bugged? There is absolutely nothing change for that half year. I again connected to robothub, cant update errors, apps not worked lol just like it was when I half year ago open box and absolutely same cant update and apps not worked 😃dd

I still hope that it can be fixed and can be usefuf…

gregflurry

MartinMotycka Once again, I much appreciate the help. I do have some followup questions, as I still have many points of confusion.

Before I get to the followup questions, I tried something new today, creating an app. I was moderately successful, but it was yet another rather painful experience. I thought it would be helpful for me, and perhaps others, to document what I did, discovered, failed at, etc. I humbly suggest that the level of detail I've tried to include could be helpful to those, like myself, that are just getting started. The document includes some more questions/mysteries, and also includes some helpful info (an appendix) on resetting and reconnection to the hub. I'd be willing to update/polish that if it would help "the cause". Sadly it is too big to attach; this is the Google drive link.

gregflurry

MartinMotycka 1. What is the proper version of the Default app to install; the candidates are 1.2.1 and 2.0.0; I have had 1.2.1 work, but I think I've tried twice to download 2.0.0 and it has failed both times

I need a lot more help with the USB-C cable stuff. I connected the RAE to my macBook via USB-C and used ssh root@192.168.197.55 with no password and that worked. I can look at the file system But now what? How would I download an app? How would I run an app? I can see various processes running, but I would not know which to kill if the need arose.
Regarding downloads, I have zero experience with Docker. How do I know if I have to download it? Would it be possible for the Hub to tell the user how big the download is going to be? Also, the Hub indicates my RAE only has 0.4 GB of "disk" available. There appears to be no way I could install an app several GB in size. Am I missing something? One thing I have found with RAE is patience is a must; everything seems to take a lot longer than one might expect.
Regarding the RAE agent, if you notice in the document I posted, you will find that I again experienced the Hub trying to update the agent. Why is that happening?
Regarding lag time, I started informally timing the lag, and it seems to be less than 2 seconds, so I guess my experience is expected.
And finally, the Default app. I have spent the last 2 hours trying to achieve what you suggest is possible and so far I am not even close. I first had to install the Default app; I had to use 1.2.1 because I could not ever get 2.0.0 to install or even download. Using the Hub, I was able to assign the device to the Default app, and was able to successfully run the app. I noticed in your post that the app was stopped, so I stopped the app; I think, anyway, because the LEDs stayed on. I fired up the agent to https://192.168.1.69:9010 (the WIFI IP address of my RAE) and when I clicked "Perception Apps" I saw something similar to you image, though my Test app was on the left and the Default app was on the right. I clicked "Open Frontend" and got a black page; I waited for a few minutes and concluded it did not work. I shutdown the device and started again and the Default app started. I tried the agent again, with the app running, and got the same results -- a blank page for a few minutes. I tried again and again, but something changed and all I got was a "twirly circle pair". This happened over several power cycles, so I gave up. Can you give me the magic incantation to get to point where I can use the Default app to manipulate the motors, LEDs, etc?

Thanks for the help.

Luxonis-Vlad

@Dominus Thanks for your post. The current state of the software stack on the RAE is definitely not final. For most of our developers, RAE was their first encounter with ROS and it allowed us to test how DepthAI integrates with ROS and how it can be used with SLAM. We realized that it is too complicated, and therefore we are now working on integrating ROS support directly into the DepthAI library, as well as SLAM/VIO. Once that is done, we will release sample applications showing how to use it. This should significantly simplify writing applications for the RAE and at the same time the use of ROS and SLAM with OAK cameras.

gregflurry

Luxonis-Vlad Thanks for the information. Reading between the lines, perhaps over-pessimistically, I see large changes in architecture, significant development, significant testing, and a significant documentation effort. Based on what I've experienced in my career as a SW and HW developer (tho I'm long retired), it could be many months before "the new RAE" is ready for those of us who want an underlying platform that is stable, works reliably with reasonable performance, facilitates fast iterative development, and is well-documented, so we can do what we want to do. Luxonis, in my opinion, achieved that quite well with the original OAK-D, which is why I was excited to get a RAE. Unfortunately, my experience with RAE so far has been quite frustrating. It is unstable, not well documented, and doing the simplest, mundane things is like paddling through molasses.

I first started trying to use RAE 5 months ago, and managed to get it into a state from which I recently found could be fixed only with a factory reset. Other tasks caused me to put RAE on the shelf. Now, 5 months later, I've spent over a week and have made almost no progress in terms of actually developing my own application. It seems to me I might be better off putting RAE back on the shelf for another 5 months, hoping the new architecture will be usable at that time. Does that make sense?

MartinMotycka

gregflurry

Hi.

Firstly, we would like to express our gratitude for your comprehensive feedback. It holds immense value for us and we truly appreciate it. We acknowledge that the user experience with RAE has room for improvement, and we want to assure you that the development of RAE is an ongoing process. As Luxonis-Vlad mentioned, there are significant enhancements planned for the future.

I will attempt to clarify some questions from your document and previous post.

I need a lot more help with the USB-C cable stuff
Apologies for any confusion. Only the firmware can be updated via USB-C. Hub applications, on the other hand, can only be installed through the Hub.

my RAE only has 0.4 GB of "disk" available
The Hub interface currently displays the system partition. We are aware of this issue and it will be addressed in a future firmware update. If you want to check free space it should be possible via SSH - df command should work.

App size
The current display shows the size of the Python script. However, when the application is downloaded, it includes much more data - the entire runtime environment. We acknowledge that this can be confusing and may lead to a poor user experience. Rest assured, this issue will be addressed in a future firmware update.

Apps development
App development in Hub involves two steps. Initially, you create an app and deploy it to the robot/device in development mode. This mode requires working on your local network, necessitating direct access to the device. All development can be done in the Robothub Studio through the web interface.

Once you've completed your work, you can publish the app to the Robothub cloud. Upon publishing, the app is built and will appear in the "Run" menu during installation. You can modify the app's name and description in the "robothub.toml" file within the Studio. It's recommended to change the name to distinguish between the development app and the launchable app.

This two-step approach offers a significant advantage when you have a fleet of devices. After you've finished development and published the app, anyone on your team can easily download and use the app. This makes deploying apps to the devices very easy. The same applies when you need to update the app. You make your changes in the code, publish a new version to the Hub, and anyone can then download the updated version.

Updating Agent
I honestly don't know why this is happen. In RAE is agent part of the firmware and it should updates only with firmware. I have to try to reproduce this behavior.

Default app
Version 1.2 requires firmware version 1.13 or earlier. Version 2.0 requires firmware version 1.14. This information should be in the installation dialog. Version 2.0 is much more bigger. So it took some time to download and install. When I have some free unit at hand, i will test how long it takes in mine environment.

Regards, M.

MartinMotycka

gregflurry
Please set sharing of the document to "everyone with link". Thanks, M.

gregflurry

MartinMotycka I madę a few changes and created a more portable PDF. Still too big, but I think I get this unrestricted access.

IvoGermann

gregflurry Hi Greg, I'm following closely your journey as I'm going through similar experiences. Can you share the document? I could not download it with either of the two links you provided. Thanks.

MartinMotycka

gregflurry
I'll review the document you provided and will get back to you shortly. Thank you for your patience. M.

gregflurry

IvoGermann I thought I made the second link public. I am traveling with limited access so it could be Tuesday before I can investigate.

gregflurry

IvoGermann By some miracle I could get to Google Drive and check. I think the file is as public as I ca make it. Here is the like Let me know what happens.

IvoGermann

gregflurry Thanks a lot for sharing this. And you are right - the 2nd link works now.