On RobotHub I see there is an update to the 'Agent Version', but whenever I click the up arrow button to "Upgrade Agent to latest version" it hangs in 'Upgrading from 23.223.1855 to 24.031.1223 / Waiting for device to connect...'. Is this currently expected?
I'm not sure if related, but I do see this in the log file /var/log/daemon.log:
2024-03-03T21:50:17.187984+00:00 rae hddl_device_service[567]: #033[1;31;40m[21:50:17.1867][567]ERROR[XLinkPlatform.cpp:88] devices_num = 1, no pcie device found, retry after 2 seconds ...#033[0m
2024-03-03T21:50:17.220529+00:00 rae robothub-tunnel.mjs[577]: {"component":null,"context":{},"level":20,"logger":{"config":{"format":"json","level":10,"rootComponent":"tunnel","runtime":"production"},"name":"tunnel"},"message":"killing process by pid 4212 (read from /run/user/1001/rh-ttyd.pid)","when":"2024-03-03T21:50:17.219Z"}
2024-03-03T21:50:17.220848+00:00 rae robothub-tunnel.mjs[577]: {"component":null,"context":{"error":{"code":"ESRCH","errno":-3,"message":"kill ESRCH","name":"Error","stack":"Error: kill ESRCH\n at process.kill (node:internal/process/per_thread:233:13)\n at killProcessByPidFile (file:///usr/libexec/robothub/robothub-tunnel.mjs:42119:15)\n at file:///usr/libexec/robothub/robothub-tunnel.mjs:42217:9\n at file:///usr/libexec/robothub/robothub-tunnel.mjs:31253:16","syscall":"kill"}},"level":40,"logger":{"config":{"format":"json","level":10,"rootComponent":"tunnel","runtime":"production"},"name":"tunnel"},"message":"kill 4212 failed","when":"2024-03-03T21:50:17.219Z"}
2024-03-03T21:50:17.222818+00:00 rae robothub-tunnel.mjs[577]: {"component":null,"context":{"args":["--interface","/home/robothub/ttyd.sock","--socket-owner","robothub:robothub","-cwd","/home/robothub","--debug","15","-t","enableZmodem=true","-t","enableTrzsz=true","-t","enableSixel=true","-t","disableLeaveAlert=true","-t","disableResizeOverlay=true","-t","fontSize=15","-t","theme={\"background\": \"#111\"}","bash"],"bin":"/usr/libexec/robothub/ttyd"},"level":30,"logger":{"config":{"format":"json","level":10,"rootComponent":"tunnel","runtime":"production"},"name":"tunnel"},"message":"starting 'ttyd'","when":"2024-03-03T21:50:17.221Z"}
2024-03-03T21:50:17.231342+00:00 rae robothub-tunnel.mjs[577]: {"component":null,"context":{},"level":30,"logger":{"config":{"format":"json","level":10,"rootComponent":"tunnel","runtime":"production"},"name":"tunnel"},"message":"'ttyd' started with PID 4216","when":"2024-03-03T21:50:17.230Z"}
2024-03-03T21:50:17.233331+00:00 rae robothub-tunnel.mjs[4216]: ttyd: invalid credential, format: username:password
2024-03-03T21:50:17.234627+00:00 rae robothub-tunnel.mjs[577]: {"component":null,"context":{"exitCode":255,"signal":null},"level":30,"logger":{"config":{"format":"json","level":10,"rootComponent":"tunnel","runtime":"production"},"name":"tunnel"},"message":"ttyd exited","when":"2024-03-03T21:50:17.233Z"}
2024-03-03T21:50:19.188682+00:00 rae hddl_device_service[567]: #033[1;31;40m[21:50:19.1873][567]ERROR[XLinkPlatform.cpp:88] devices_num = 1, no pcie device found, retry after 2 seconds ...#033[0m
This error occurs regardless of which app runs (though, I have yet to get any app to work other than the default app)
When clicking update, robothub-ctl logs says:
...
level=info msg="Loaded configuration file: /var/lib/mender/mender.conf" module=config
level=warning msg="No server URL(s) specified in mender configuration." module=config
level=warning msg="Server entry 1 has no associated server URL." module=config
level=info msg="Mender running on partition: /dev/mmcblk1p8" module=main
level=info msg="Performing remote update from: [https://luxonisos.fra1.digitaloceanspaces.com/build/1.14-rae+6263cb8c60f2cbfc80cc523f2b37581eb38139aa/luxonisos-1.14-rae.dm-verity.mender]." module=standalone
level=info msg="no public key was provided for authenticating the artifact" module=installer
level=info msg="Update module output: Updating rootfs: /dev/mmcblk1p5" module=modules
... minutes later ...
level=info msg="Update module output: Updating rootfs: /dev/mmcblk1p6" module=modules
level=error msg="Download connection broken: stream error: stream ID 1; NO_ERROR" module=update_resumer
level=info msg="Resuming download in 1m0s" module=update_resumer
... a little later ...
level=info msg="Attempting to resume artifact download from offset 867356933" module=update_resumer
63 22:01:55 I root/agent/root : firmware upgrade> upgrade was successful, rebooting after 5 seconds
{
"code": 0
}
... rae reboots, on reconnect ...
63 22:03:04 I user/agent : user: robothub
63 22:03:04 I user/agent/hub/config : startup cloud state
{
"connectionConfig": {
"remoteHost": "robothub.luxonis.com",
"remoteProto": "ssl",
"remoteWebPort": null
},
"decodedToken": null,
"team": null,
"token": "present",
"webUri": "https://robothub.luxonis.com"
}
63 22:03:05 W user/agent/local-mqtt : cant connect to mqtt, is broker running?
63 22:03:06 W user/agent/local-mqtt : cant connect to mqtt, is broker running?
63 22:03:06 E user/agent/apps : failed to kill container b8028d63c54efbbed09dc0d71d7ea6ebc58ce2d85f9f27dc6a020c7230f2cec9: Error: killContainer exited with 125
63 22:03:06 I agent/nanomq : starting nanomq process...
63 22:03:06 I user/agent/hub-connection : no longer calling home (waiting if token removed)
63 22:03:06 I user/agent/hub-connection : MQTT connection -> connecting... wss://gateway.robothub.luxonis.com
Sun, 03 Mar 2024 22:03:06 GMT body-parser deprecated undefined extended: provide extended option at node:internal/process/task_queues:60:5
63 22:03:06 I user/agent/http-server : HTTP Server listening on 127.0.0.1:9009
63 22:03:06 I user/agent/http-server : HTTP Server listening on 0.0.0.0:9010
63 22:03:06 I agent/nanomq : NanoMQ Broker is started successfully!
63 22:03:06 I agent/nanomq : 2024-03-03 22:03:06 [805] ERROR /home/runner/work/nanomq/nanomq/nng/src/sp/protocol/mqtt/auth_http.c:281 send_request: Connect failed: Connection refused
63 22:03:06 I agent/nanomq :
63 22:03:06 I agent/nanomq : 2024-03-03 22:03:06 [805] WARN /home/runner/work/nanomq/nanomq/nng/src/sp/protocol/mqtt/nmq_mqtt.c:728 nano_pipe_start: Invalid auth info.
63 22:03:06 I agent/nanomq : 2024-03-03 22:03:06 [805] WARN /home/runner/work/nanomq/nanomq/nng/src/sp/transport/mqtt/broker_tcp.c:610 tcptran_pipe_recv_cb: nni aio recv error!! Object closed
63 22:03:06 I agent/nanomq :
63 22:03:06 I agent/nanomq : 2024-03-03 22:03:06 [805] WARN /home/runner/work/nanomq/nanomq/nng/src/sp/transport/mqtt/broker_tcp.c:872 tcptran_pipe_recv_cb: tcptran_pipe_recv_cb: recv error rv: 139
63 22:03:06 I agent/nanomq :
63 22:03:06 I agent/nanomq : 2024-03-03 22:03:06 [808] ERROR /home/runner/work/nanomq/nanomq/nng/src/sp/protocol/mqtt/auth_http.c:281 send_request: Connect failed: Connection refused
63 22:03:06 I agent/nanomq :
63 22:03:06 I agent/nanomq : 2024-03-03 22:03:06 [808] WARN /home/runner/work/nanomq/nanomq/nng/src/sp/protocol/mqtt/nmq_mqtt.c:728 nano_pipe_start: Invalid auth info.
63 22:03:06 I agent/nanomq : 2024-03-03 22:03:06 [808] WARN /home/runner/work/nanomq/nanomq/nng/src/sp/transport/mqtt/broker_tcp.c:610 tcptran_pipe_recv_cb: nni aio recv error!! Object closed
63 22:03:06 I agent/nanomq :
63 22:03:06 I agent/nanomq : 2024-03-03 22:03:06 [808] WARN /home/runner/work/nanomq/nanomq/nng/src/sp/transport/mqtt/broker_tcp.c:872 tcptran_pipe_recv_cb: tcptran_pipe_recv_cb: recv error rv: 139
63 22:03:06 I agent/nanomq :
63 22:03:06 I agent/nanomq :
63 22:03:06 W user/agent/hub-connection : network error (connection-refused) | address: undefined
63 22:03:06 E root/agent/root : mqtt error
{
"error": {
"code": 135,
"message": "Connection refused: Not authorized",
"name": "Error",
"stack": "Error: Connection refused: Not authorized\n at MqttClient._handleConnack (file:///usr/libexec/robothub/robothub-agent.mjs:38517:21)\n at MqttClient._handlePacket (file:///usr/libexec/robothub/robothub-agent.mjs:37916:16)\n at work (file:///usr/libexec/robothub/robothub-agent.mjs:37822:16)\n at writable._write (file:///usr/libexec/robothub/robothub-agent.mjs:37837:9)\n at doWrite (file:///usr/libexec/robothub/robothub-agent.mjs:27194:16)\n at writeOrBuffer (file:///usr/libexec/robothub/robothub-agent.mjs:27179:9)\n at Writable.write (file:///usr/libexec/robothub/robothub-agent.mjs:27101:15)\n at Socket.ondata (node:internal/streams/readable:766:22)\n at Socket.emit (node:events:513:28)\n at Socket.emit (node:domain:489:12)"
}
}
63 22:03:07 I user/agent/apps : local sync (db -> containers) - assigning devices to apps...
63 22:03:07 I user/agent/apps : local sync (db -> containers) - assigning orphan devices to builtin apps...
{
"device": [
"3C52A016838860C5"
]
}
63 22:03:07 I user/agent/apps : local sync (db -> containers) - assigning orphan devices to builtin apps...
{
"device": [
"3C52A016838860C5"
]
}
63 22:03:07 I stream-handler : PID: 818
63 22:03:07 I stream-handler : loading bootstrap configuration from "/etc/robothub-agent/agent.toml"
63 22:03:07 I user/agent/apps : local sync (db -> containers) | storedApps = 1 | containers = 1
63 22:03:07 I user/agent/apps : local sync (db -> containers) - ensure stored containers...
63 22:03:07 I stream-handler/rtc-connection : configured
63 22:03:08 I user/agent/apps : expected app[117fc4de-c8c9-4361-895f-1a43be8f1eed] to be running, but its stopped -> starting...
63 22:03:09 I agent/nanomq : 2024-03-03 22:03:09 [805] WARN /home/runner/work/nanomq/nanomq/nng/src/sp/transport/mqtt/broker_tcp.c:610 tcptran_pipe_recv_cb: nni aio recv error!! Connection shutdown
63 22:03:09 I agent/nanomq :
63 22:03:09 I agent/nanomq : 2024-03-03 22:03:09 [805] WARN /home/runner/work/nanomq/nanomq/nng/src/sp/transport/mqtt/broker_tcp.c:872 tcptran_pipe_recv_cb: tcptran_pipe_recv_cb: recv error rv: 139
63 22:03:09 I agent/nanomq :
63 22:03:09 I agent/nanomq :
63 22:03:11 I agent/nanomq : 2024-03-03 22:03:11 [808] WARN /home/runner/work/nanomq/nanomq/nng/src/sp/transport/mqtt/broker_tcp.c:610 tcptran_pipe_recv_cb: nni aio recv error!! Connection shutdown
63 22:03:11 I agent/nanomq :
63 22:03:11 I agent/nanomq : 2024-03-03 22:03:11 [808] WARN /home/runner/work/nanomq/nanomq/nng/src/sp/transport/mqtt/broker_tcp.c:872 tcptran_pipe_recv_cb: tcptran_pipe_recv_cb: recv error rv: 139
63 22:03:11 I agent/nanomq :
63 22:03:11 I agent/nanomq :
63 22:03:11 I user/agent/apps : local sync (db -> containers) - cleanup of old containers...
63 22:03:11 I user/agent/apps : local sync (db -> containers) - cleanup of old containers...
63 22:03:11 I user/agent/apps : local sync (db -> containers) - done
63 22:03:12 I user/agent/local-mqtt : [117fc4de-c8c9-4361-895f-1a43be8f1eed] -> 63 22:03:12 D inner/mqtt-client : subscription-change -> connected, fromConnect = false
63 22:03:12 I user/agent/local-mqtt : [117fc4de-c8c9-4361-895f-1a43be8f1eed] -> 63 22:03:12 D inner/mqtt-client : subscribed to app/117fc4de-c8c9-4361-895f-1a43be8f1eed/inner
63 22:03:12 I user/agent/local-mqtt : [117fc4de-c8c9-4361-895f-1a43be8f1eed] -> 63 22:03:12 D inner/mqtt-client : subscribed to app/117fc4de-c8c9-4361-895f-1a43be8f1eed/forward
63 22:03:12 I user/agent/local-mqtt : [117fc4de-c8c9-4361-895f-1a43be8f1eed] -> 63 22:03:12 D inner/mqtt-client : subscribed to app/117fc4de-c8c9-4361-895f-1a43be8f1eed/+/fe
63 22:03:12 I user/agent/local-mqtt : [117fc4de-c8c9-4361-895f-1a43be8f1eed] -> 63 22:03:12 D inner : respawning app
63 22:03:12 I user/agent/local-mqtt : [117fc4de-c8c9-4361-895f-1a43be8f1eed] -> 63 22:03:12 D inner/mqtt-client : subscription-change -> connected, fromConnect = false
63 22:03:13 I user/agent/local-mqtt : [117fc4de-c8c9-4361-895f-1a43be8f1eed] -> 63 22:03:13 I inner : [2024-03-03 22:03:13.266] [depthai] [warning] USB protocol not available - If running in a container, make sure that the following is set: "-v /dev/bus/usb:/dev/bus/usb --device-cgroup-rule='c 189:* rmw'"
63 22:03:13 I user/agent/local-mqtt : [117fc4de-c8c9-4361-895f-1a43be8f1eed] -> 63 22:03:13 D inner : app status changed to: running
63 22:03:13 I user/agent/local-mqtt : [117fc4de-c8c9-4361-895f-1a43be8f1eed] -> 63 22:03:13 D inner/mqtt-client : subscription-change -> connected, fromConnect = true
63 22:03:16 I user/agent/hub-connection : MQTT connection -> connecting... wss://gateway.robothub.luxonis.com
63 22:03:18 W user/agent/hub-connection : Last agent upgrade probably failed
63 22:03:24 I user/agent/hub-connection : received see_you
63 22:03:24 I user/agent/hub-connection : Connection with server established, communication enabled!
The mender command from this post to downloaded the update manually does the same thing.
