Agent Version update working?

QQuantumNature · Mar 3, 2024

On RobotHub I see there is an update to the 'Agent Version', but whenever I click the up arrow button to "Upgrade Agent to latest version" it hangs in 'Upgrading from 23.223.1855 to 24.031.1223 / Waiting for device to connect...'. Is this currently expected?

I'm not sure if related, but I do see this in the log file /var/log/daemon.log:

2024-03-03T21:50:17.187984+00:00 rae hddl_device_service[567]: #033[1;31;40m[21:50:17.1867][567]ERROR[XLinkPlatform.cpp:88] devices_num = 1, no pcie device found, retry after 2 seconds ...#033[0m
2024-03-03T21:50:17.220529+00:00 rae robothub-tunnel.mjs[577]: {"component":null,"context":{},"level":20,"logger":{"config":{"format":"json","level":10,"rootComponent":"tunnel","runtime":"production"},"name":"tunnel"},"message":"killing process by pid 4212 (read from /run/user/1001/rh-ttyd.pid)","when":"2024-03-03T21:50:17.219Z"}
2024-03-03T21:50:17.220848+00:00 rae robothub-tunnel.mjs[577]: {"component":null,"context":{"error":{"code":"ESRCH","errno":-3,"message":"kill ESRCH","name":"Error","stack":"Error: kill ESRCH\n    at process.kill (node:internal/process/per_thread:233:13)\n    at killProcessByPidFile (file:///usr/libexec/robothub/robothub-tunnel.mjs:42119:15)\n    at file:///usr/libexec/robothub/robothub-tunnel.mjs:42217:9\n    at file:///usr/libexec/robothub/robothub-tunnel.mjs:31253:16","syscall":"kill"}},"level":40,"logger":{"config":{"format":"json","level":10,"rootComponent":"tunnel","runtime":"production"},"name":"tunnel"},"message":"kill 4212 failed","when":"2024-03-03T21:50:17.219Z"}
2024-03-03T21:50:17.222818+00:00 rae robothub-tunnel.mjs[577]: {"component":null,"context":{"args":["--interface","/home/robothub/ttyd.sock","--socket-owner","robothub:robothub","-cwd","/home/robothub","--debug","15","-t","enableZmodem=true","-t","enableTrzsz=true","-t","enableSixel=true","-t","disableLeaveAlert=true","-t","disableResizeOverlay=true","-t","fontSize=15","-t","theme={\"background\": \"#111\"}","bash"],"bin":"/usr/libexec/robothub/ttyd"},"level":30,"logger":{"config":{"format":"json","level":10,"rootComponent":"tunnel","runtime":"production"},"name":"tunnel"},"message":"starting 'ttyd'","when":"2024-03-03T21:50:17.221Z"}
2024-03-03T21:50:17.231342+00:00 rae robothub-tunnel.mjs[577]: {"component":null,"context":{},"level":30,"logger":{"config":{"format":"json","level":10,"rootComponent":"tunnel","runtime":"production"},"name":"tunnel"},"message":"'ttyd' started with PID 4216","when":"2024-03-03T21:50:17.230Z"}
2024-03-03T21:50:17.233331+00:00 rae robothub-tunnel.mjs[4216]: ttyd: invalid credential, format: username:password
2024-03-03T21:50:17.234627+00:00 rae robothub-tunnel.mjs[577]: {"component":null,"context":{"exitCode":255,"signal":null},"level":30,"logger":{"config":{"format":"json","level":10,"rootComponent":"tunnel","runtime":"production"},"name":"tunnel"},"message":"ttyd exited","when":"2024-03-03T21:50:17.233Z"}
2024-03-03T21:50:19.188682+00:00 rae hddl_device_service[567]: #033[1;31;40m[21:50:19.1873][567]ERROR[XLinkPlatform.cpp:88] devices_num = 1, no pcie device found, retry after 2 seconds ...#033[0m

This error occurs regardless of which app runs (though, I have yet to get any app to work other than the default app)

When clicking update, robothub-ctl logs says:

...
level=info msg="Loaded configuration file: /var/lib/mender/mender.conf" module=config
level=warning msg="No server URL(s) specified in mender configuration." module=config
level=warning msg="Server entry 1 has no associated server URL." module=config
level=info msg="Mender running on partition: /dev/mmcblk1p8" module=main
level=info msg="Performing remote update from: [https://luxonisos.fra1.digitaloceanspaces.com/build/1.14-rae+6263cb8c60f2cbfc80cc523f2b37581eb38139aa/luxonisos-1.14-rae.dm-verity.mender]." module=standalone
level=info msg="no public key was provided for authenticating the artifact" module=installer
level=info msg="Update module output: Updating rootfs: /dev/mmcblk1p5" module=modules
... minutes later ...
level=info msg="Update module output: Updating rootfs: /dev/mmcblk1p6" module=modules
level=error msg="Download connection broken: stream error: stream ID 1; NO_ERROR" module=update_resumer
level=info msg="Resuming download in 1m0s" module=update_resumer
... a little later ...
level=info msg="Attempting to resume artifact download from offset 867356933" module=update_resumer
63 22:01:55  I  root/agent/root : firmware upgrade> upgrade was successful, rebooting after 5 seconds
{
  "code": 0
}
... rae reboots, on reconnect ...
63 22:03:04  I  user/agent : user: robothub
63 22:03:04  I  user/agent/hub/config : startup cloud state
{
  "connectionConfig": {
    "remoteHost": "robothub.luxonis.com",
    "remoteProto": "ssl",
    "remoteWebPort": null
  },
  "decodedToken": null,
  "team": null,
  "token": "present",
  "webUri": "https://robothub.luxonis.com"
}
63 22:03:05  W  user/agent/local-mqtt : cant connect to mqtt, is broker running?
63 22:03:06  W  user/agent/local-mqtt : cant connect to mqtt, is broker running?
63 22:03:06  E  user/agent/apps : failed to kill container b8028d63c54efbbed09dc0d71d7ea6ebc58ce2d85f9f27dc6a020c7230f2cec9: Error: killContainer exited with 125
63 22:03:06  I  agent/nanomq : starting nanomq process...
63 22:03:06  I  user/agent/hub-connection : no longer calling home (waiting if token removed)
63 22:03:06  I  user/agent/hub-connection : MQTT connection -> connecting... wss://gateway.robothub.luxonis.com
Sun, 03 Mar 2024 22:03:06 GMT body-parser deprecated undefined extended: provide extended option at node:internal/process/task_queues:60:5
63 22:03:06  I  user/agent/http-server : HTTP Server listening on 127.0.0.1:9009
63 22:03:06  I  user/agent/http-server : HTTP Server listening on 0.0.0.0:9010
63 22:03:06  I  agent/nanomq : NanoMQ Broker is started successfully!
63 22:03:06  I  agent/nanomq : 2024-03-03 22:03:06 [805] ERROR /home/runner/work/nanomq/nanomq/nng/src/sp/protocol/mqtt/auth_http.c:281 send_request: Connect failed: Connection refused
63 22:03:06  I  agent/nanomq :
63 22:03:06  I  agent/nanomq : 2024-03-03 22:03:06 [805] WARN  /home/runner/work/nanomq/nanomq/nng/src/sp/protocol/mqtt/nmq_mqtt.c:728 nano_pipe_start: Invalid auth info.
63 22:03:06  I  agent/nanomq : 2024-03-03 22:03:06 [805] WARN  /home/runner/work/nanomq/nanomq/nng/src/sp/transport/mqtt/broker_tcp.c:610 tcptran_pipe_recv_cb: nni aio recv error!! Object closed
63 22:03:06  I  agent/nanomq :
63 22:03:06  I  agent/nanomq : 2024-03-03 22:03:06 [805] WARN  /home/runner/work/nanomq/nanomq/nng/src/sp/transport/mqtt/broker_tcp.c:872 tcptran_pipe_recv_cb: tcptran_pipe_recv_cb: recv error rv: 139
63 22:03:06  I  agent/nanomq :
63 22:03:06  I  agent/nanomq : 2024-03-03 22:03:06 [808] ERROR /home/runner/work/nanomq/nanomq/nng/src/sp/protocol/mqtt/auth_http.c:281 send_request: Connect failed: Connection refused
63 22:03:06  I  agent/nanomq :
63 22:03:06  I  agent/nanomq : 2024-03-03 22:03:06 [808] WARN  /home/runner/work/nanomq/nanomq/nng/src/sp/protocol/mqtt/nmq_mqtt.c:728 nano_pipe_start: Invalid auth info.
63 22:03:06  I  agent/nanomq : 2024-03-03 22:03:06 [808] WARN  /home/runner/work/nanomq/nanomq/nng/src/sp/transport/mqtt/broker_tcp.c:610 tcptran_pipe_recv_cb: nni aio recv error!! Object closed
63 22:03:06  I  agent/nanomq :
63 22:03:06  I  agent/nanomq : 2024-03-03 22:03:06 [808] WARN  /home/runner/work/nanomq/nanomq/nng/src/sp/transport/mqtt/broker_tcp.c:872 tcptran_pipe_recv_cb: tcptran_pipe_recv_cb: recv error rv: 139
63 22:03:06  I  agent/nanomq :
63 22:03:06  I  agent/nanomq :
63 22:03:06  W  user/agent/hub-connection : network error (connection-refused) | address: undefined
63 22:03:06  E  root/agent/root : mqtt error
{
  "error": {
    "code": 135,
    "message": "Connection refused: Not authorized",
    "name": "Error",
    "stack": "Error: Connection refused: Not authorized\n    at MqttClient._handleConnack (file:///usr/libexec/robothub/robothub-agent.mjs:38517:21)\n    at MqttClient._handlePacket (file:///usr/libexec/robothub/robothub-agent.mjs:37916:16)\n    at work (file:///usr/libexec/robothub/robothub-agent.mjs:37822:16)\n    at writable._write (file:///usr/libexec/robothub/robothub-agent.mjs:37837:9)\n    at doWrite (file:///usr/libexec/robothub/robothub-agent.mjs:27194:16)\n    at writeOrBuffer (file:///usr/libexec/robothub/robothub-agent.mjs:27179:9)\n    at Writable.write (file:///usr/libexec/robothub/robothub-agent.mjs:27101:15)\n    at Socket.ondata (node:internal/streams/readable:766:22)\n    at Socket.emit (node:events:513:28)\n    at Socket.emit (node:domain:489:12)"
  }
}
63 22:03:07  I  user/agent/apps : local sync (db -> containers) - assigning devices to apps...
63 22:03:07  I  user/agent/apps : local sync (db -> containers) - assigning orphan devices to builtin apps...
{
  "device": [
    "3C52A016838860C5"
  ]
}
63 22:03:07  I  user/agent/apps : local sync (db -> containers) - assigning orphan devices to builtin apps...
{
  "device": [
    "3C52A016838860C5"
  ]
}
63 22:03:07  I  stream-handler : PID: 818
63 22:03:07  I  stream-handler : loading bootstrap configuration from "/etc/robothub-agent/agent.toml"
63 22:03:07  I  user/agent/apps : local sync (db -> containers) | storedApps = 1 | containers = 1
63 22:03:07  I  user/agent/apps : local sync (db -> containers) - ensure stored containers...
63 22:03:07  I  stream-handler/rtc-connection : configured
63 22:03:08  I  user/agent/apps : expected app[117fc4de-c8c9-4361-895f-1a43be8f1eed] to be running, but its stopped -> starting...
63 22:03:09  I  agent/nanomq : 2024-03-03 22:03:09 [805] WARN  /home/runner/work/nanomq/nanomq/nng/src/sp/transport/mqtt/broker_tcp.c:610 tcptran_pipe_recv_cb: nni aio recv error!! Connection shutdown
63 22:03:09  I  agent/nanomq :
63 22:03:09  I  agent/nanomq : 2024-03-03 22:03:09 [805] WARN  /home/runner/work/nanomq/nanomq/nng/src/sp/transport/mqtt/broker_tcp.c:872 tcptran_pipe_recv_cb: tcptran_pipe_recv_cb: recv error rv: 139
63 22:03:09  I  agent/nanomq :
63 22:03:09  I  agent/nanomq :
63 22:03:11  I  agent/nanomq : 2024-03-03 22:03:11 [808] WARN  /home/runner/work/nanomq/nanomq/nng/src/sp/transport/mqtt/broker_tcp.c:610 tcptran_pipe_recv_cb: nni aio recv error!! Connection shutdown
63 22:03:11  I  agent/nanomq :
63 22:03:11  I  agent/nanomq : 2024-03-03 22:03:11 [808] WARN  /home/runner/work/nanomq/nanomq/nng/src/sp/transport/mqtt/broker_tcp.c:872 tcptran_pipe_recv_cb: tcptran_pipe_recv_cb: recv error rv: 139
63 22:03:11  I  agent/nanomq :
63 22:03:11  I  agent/nanomq :
63 22:03:11  I  user/agent/apps : local sync (db -> containers) - cleanup of old containers...
63 22:03:11  I  user/agent/apps : local sync (db -> containers) - cleanup of old containers...
63 22:03:11  I  user/agent/apps : local sync (db -> containers) - done
63 22:03:12  I  user/agent/local-mqtt : [117fc4de-c8c9-4361-895f-1a43be8f1eed] -> 63 22:03:12  D  inner/mqtt-client : subscription-change -> connected, fromConnect = false
63 22:03:12  I  user/agent/local-mqtt : [117fc4de-c8c9-4361-895f-1a43be8f1eed] -> 63 22:03:12  D  inner/mqtt-client : subscribed to app/117fc4de-c8c9-4361-895f-1a43be8f1eed/inner
63 22:03:12  I  user/agent/local-mqtt : [117fc4de-c8c9-4361-895f-1a43be8f1eed] -> 63 22:03:12  D  inner/mqtt-client : subscribed to app/117fc4de-c8c9-4361-895f-1a43be8f1eed/forward
63 22:03:12  I  user/agent/local-mqtt : [117fc4de-c8c9-4361-895f-1a43be8f1eed] -> 63 22:03:12  D  inner/mqtt-client : subscribed to app/117fc4de-c8c9-4361-895f-1a43be8f1eed/+/fe
63 22:03:12  I  user/agent/local-mqtt : [117fc4de-c8c9-4361-895f-1a43be8f1eed] -> 63 22:03:12  D  inner : respawning app
63 22:03:12  I  user/agent/local-mqtt : [117fc4de-c8c9-4361-895f-1a43be8f1eed] -> 63 22:03:12  D  inner/mqtt-client : subscription-change -> connected, fromConnect = false
63 22:03:13  I  user/agent/local-mqtt : [117fc4de-c8c9-4361-895f-1a43be8f1eed] -> 63 22:03:13  I  inner : [2024-03-03 22:03:13.266] [depthai] [warning] USB protocol not available - If running in a container, make sure that the following is set: "-v /dev/bus/usb:/dev/bus/usb --device-cgroup-rule='c 189:* rmw'"
63 22:03:13  I  user/agent/local-mqtt : [117fc4de-c8c9-4361-895f-1a43be8f1eed] -> 63 22:03:13  D  inner : app status changed to: running
63 22:03:13  I  user/agent/local-mqtt : [117fc4de-c8c9-4361-895f-1a43be8f1eed] -> 63 22:03:13  D  inner/mqtt-client : subscription-change -> connected, fromConnect = true
63 22:03:16  I  user/agent/hub-connection : MQTT connection -> connecting... wss://gateway.robothub.luxonis.com
63 22:03:18  W  user/agent/hub-connection : Last agent upgrade probably failed
63 22:03:24  I  user/agent/hub-connection : received see_you
63 22:03:24  I  user/agent/hub-connection : Connection with server established, communication enabled!

The mender command from this post to downloaded the update manually does the same thing.

jakaskerl · Mar 4, 2024

Hi @QuantumNature
Likely a bug in the frontend. Any chance the robot is offline when you are attempting the upgrade? Could you provide the team ID so we can check.

Thanks,
Jaka

QQuantumNature · Mar 4, 2024

Robot was definitely online. And team ID? Team name is "Pink", is that what you need to ID my Rae?

Rriperiherino · Mar 4, 2024

Same issue.

Is this product even maintained?

Getting Started - https://docs-beta.luxonis.com/hardware/rae/get-started/ (Unexpected server error!)

RAE SDK - https://docs-beta.luxonis.com/develop/rae/rae-sdk/ (404 / Not Found)

RAE ROS - https://docs-beta.luxonis.com/develop/rae/rae-ros/ (404 / Not Found)

Updating Firmware - https://docs-beta.luxonis.com/hardware/rae/firmware/(Unexpected server error!)

jakaskerl · Mar 5, 2024

QuantumNature Robot was definitely online. And team ID? Team name is "Pink", is that what you need to ID my Rae?

There is actually no new release for Rae OS. It was a bug in the frontend and should now be fixed.

riperiherino Is this product even maintained?

Getting Started - https://docs-beta.luxonis.com/hardware/rae/get-started/ (Unexpected server error!)

RAE SDK - https://docs-beta.luxonis.com/develop/rae/rae-sdk/ (404 / Not Found)

RAE ROS - https://docs-beta.luxonis.com/develop/rae/rae-ros/ (404 / Not Found)

Updating Firmware - https://docs-beta.luxonis.com/hardware/rae/firmware/(Unexpected server error!)

Docs have been refactored, so some old links didn't work. Should all be working now, but the links are different.

Thanks,
Jaka

QQuantumNature · Mar 6, 2024

Lovely, thanks for the update. The up arrow is now gone and no longer tempting me into non-existent updates

And good to see some of the links back up; especially https://docs-beta.luxonis.com/hardware/rae/rae-sdk/