Bad gateway error on Hub

oakleydoakley

We have a Oak4D camera and are using the free version of the hub that is limited to five devices. The hub icon is now visible in oak viewer. However, when publishing an app to the hub, we receive the following error message:

C:\Users\james.yu\oak_test\oak-inspection-app> oakctl hub publish com-inspection-oak-eval_0-0-1.oakapp
Publishing com-inspection-oak-eval_0-0-1.oakapp
⠙ [00:00:12] [███████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] 470.29 MiB/955.23 MiB (19s)oakctl command failed:
0: Hub service at 'https://apps-ingest.cloud.luxonis.com/v1' encountered an error: Unexpected response with status code: 502 Bad Gateway with text: Some(
"Bad Gateway",
)
1: Unexpected response with status code: 502 Bad Gateway with text: Some(
"Bad Gateway",
)

What is cause of this error?

OskarSonc

oakleydoakley - this error is most likely not related to the free Hub 5-device limit. The 502 Bad Gateway is coming from the Hub app upload service during oakctl hub publish, so the usual causes are an outdated oakctl version, a transient Hub-side upload issue, or the upload failing because the .oakapp is relatively large. One known platform-side limit is that very long uploads can hit a hard timeout of about 1 hour, but in your logs it says 19s if that is correct?

Please try:

updating to the latest oakctl
rebuilding and publishing again
if possible, reducing the .oakapp size before retrying

If it still fails, please share your oakctl version, the size of the generated .oakapp file, and roughly how long the upload runs before failing.

Thanks,
Oskar

oakleydoakley

I've updated oakctl to 0.20.0. The size of the app is 800mb. The hub is showing I've already used up almost 20gb of the free storage but I don't have any apps loaded so don't know how that's possible.

OskarSonc

oakleydoakley - the storage usage is the more likely issue here. We’ve seen cases where Hub quota was consumed even when no obvious apps or data were visible, for example when a device was still trying to upload events/images in the background or when failed uploads left temporary storage allocations behind.

Could you please check:

whether any device in the workspace, including offline ones, still has apps running
whether any app has automatic image/event upload enabled
the Team Apps page for any failed or partial uploads

Also, you could create a new free profile and test it there if you don't find any above?

Thanks,
Oskar

oakleydoakley

Thanks Oskar, here is another update:

Problem started when trying to access the camera app frontend live view at 10.10.3.6:9000 and 10.10.3.7:9000. The page will load but show a blank white screen

Thesee are apps that have been installed to both cameras a week before and have been running fine until now.

Upon inspecting app logs using oakctl app logs, there are a lot of these out of space errors:

[2026-05-27T01:43:56Z] 2026/05/27 01:43:56 [crit] 36#36: *49 mkdir() "/var/lib/nginx/proxy/2" failed (28: No space left on device) while reading upstream, client: 10.10.3.98, server: _, request: "GET /assets/index-CV3zlxZC.js HTTP/1.1", upstream: "http://127.0.0.1:8082/assets/index-CV3zlxZC.js", host: "10.10.3.7:9000", referrer: "https://10.10.3.7:9000/"

This was the only app installed on the device. I followed the Luxonis guide online and uninstalled the app and pruned the storage using oakctl:

PS P:\> oakctl app prune-storage

✔ Storage pruned successfully.

PS P:\> oakctl app list

No apps installed

I then attempted to reinstall the app but got this error:

PS C:\Users\james.yu\oak_test\oak-inspection-app> oakctl app install .\com-inspection-oak-eval_0-0-3.oakapp

✖ Failed to send data.

        oakctl command failed:

0: File sync sender failed to propagate response: channel closed

Then I attempted to run the app in dev mode from the computer:

PS C:\Users\james.yu\oak_test\oak-inspection-app> oakctl app run .

Starting oakapp in dev mode from path: .

✔ Data sent to device ✔ Develop container started Press Ctrl+C to stop the app ...

App develop build output:

Retrieving base image luxonis/oakapp-base:1.2.5 ...

Layer downloaded 19ad5c44aca2 [████████████████████████████████████████] 100% (725 B)

Layer downloaded c4ade8df7c0d [████████████████████████████████████████] 100% (1009 B)

Layer downloaded adf5ff65dba2 [████████████████████████████████████████] 100% (87.99 MiB)

Layer downloaded ef5f92ac487a [████████████████████████████████████████] 100% (1.65 KiB)

Layer downloaded dff73aeeb436 [████████████████████████████████████████] 100% (197 B)

Layer downloaded 30dd576f8fce [████████████████████████████████████████] 100% (97.80 MiB)

Layer downloaded 37259e733066 [████████████████████████████████████████] 100% (26.78 MiB)

Layer downloaded b81191b31534 [████████████████████████████████████████] 100% (523 B)

Layer downloaded c844e6c88a6a [████████████████████████████████████████] 100% (6.78 MiB)

Layer downloaded f800ae88b1c8 [████████████████████████████████████████] 100% (2.25 KiB)

Layer downloaded 3c0bc210f0ed [████████████████████████████████████████] 100% (551 B)

Layer downloaded c2c19c43459d [████████████████████████████████████████] 100% (421 B)

Layer downloaded a6889824e6ef [████████████████████████████████████████] 100% (53.90 MiB) Processing layer zoo_models ...

Processing layer runtime ...

Processing layer build ...

Processing layer frontend ...

Processing layer app ...

[2026-05-27T01:43:26Z] Collecting depthai

[2026-05-27T01:43:26Z] Downloading depthai-3.6.1-cp39.cp310.cp311.cp312.cp313.cp314-cp39.cp310.cp311.cp312.cp313.cp314-manylinux_2_28_aarch64.whl.metadata (9.9 kB)

[2026-05-27T01:43:26Z] Collecting numpy<3.0.0 (from depthai)

[2026-05-27T01:43:26Z] Downloading numpy-2.4.6-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl.metadata (6.6 kB)

[2026-05-27T01:43:27Z] Downloading depthai-3.6.1-cp39.cp310.cp311.cp312.cp313.cp314-cp39.cp310.cp311.cp312.cp313.cp314-manylinux_2_28_aarch64.whl (76.9 MB)

[2026-05-27T01:43:33Z] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 76.9/76.9 MB 5.4 MB/s eta 0:00:00

[2026-05-27T01:43:33Z] Downloading numpy-2.4.6-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl (15.7 MB)

[2026-05-27T01:43:34Z] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.7/15.7 MB 12.7 MB/s eta 0:00:00

[2026-05-27T01:43:34Z] Installing collected packages: numpy, depthai

[2026-05-27T01:43:35Z] WARNING: The scripts f2py and numpy-config are installed in '/usr/local/python3.12/bin' which is not on PATH.

[2026-05-27T01:43:35Z] Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

[2026-05-27T01:43:37Z] WARNING: The script depthai is installed in '/usr/local/python3.12/bin' which is not on PATH.

[2026-05-27T01:43:37Z] Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

[2026-05-27T01:43:38Z] Successfully installed depthai-3.6.1 numpy-2.4.6

[2026-05-27T01:43:38Z] WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[2026-05-27T01:43:38Z]

[2026-05-27T01:43:38Z] [notice] A new release of pip is available: 24.0 -> 26.1.1

[2026-05-27T01:43:38Z] [notice] To update, run: python3.12 -m pip install --upgrade pip

[2026-05-27T01:43:38Z]

[2026-05-27T01:43:39Z]

App output:

[2026-05-27T01:43:41Z] OAK_WEBRTC - Starting

[2026-05-27T01:43:41Z] NGINX - Starting

[2026-05-27T01:43:41Z] find: '': No such file or directory

[2026-05-27T01:43:41Z] CONNECTION - System time is: Wed May 27 01:43:41 UTC 2026, starting the application

[2026-05-27T01:43:41Z] NGINX - Server configuration:

[2026-05-27T01:43:41Z] [2026-05-27T01:43:41Z INFO oak_webrtc] Creating authorized WebRTC client client_id=a2a6a565-057f-4590-a91b-45d2738dd729

[2026-05-27T01:43:41Z] server {

[2026-05-27T01:43:41Z] listen 9000 ssl;

[2026-05-27T01:43:41Z] listen [::]:9000 ssl;

[2026-05-27T01:43:41Z]

[2026-05-27T01:43:41Z] ssl_certificate /etc/nginx/ssl/nginx-selfsigned.crt;

[2026-05-27T01:43:41Z] ssl_certificate_key /etc/nginx/ssl/nginx-selfsigned.key;

[2026-05-27T01:43:41Z]

[2026-05-27T01:43:41Z] error_page 497 301 =307 https://$host:$server_port$request_uri;

[2026-05-27T01:43:41Z]

[2026-05-27T01:43:41Z] location /dai {

[2026-05-27T01:43:41Z] proxy_pass http://127.0.0.1:8765;

[2026-05-27T01:43:41Z] proxy_http_version 1.1;

[2026-05-27T01:43:41Z] proxy_set_header Upgrade $http_upgrade;

[2026-05-27T01:43:41Z] proxy_set_header Connection "upgrade";

[2026-05-27T01:43:41Z] proxy_set_header Host $host;

[2026-05-27T01:43:41Z] proxy_set_header X-Real-IP $remote_addr;

[2026-05-27T01:43:41Z] proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

[2026-05-27T01:43:41Z] }

[2026-05-27T01:43:41Z]

[2026-05-27T01:43:41Z] location / {

[2026-05-27T01:43:41Z]

[2026-05-27T01:43:41Z] proxy_pass http://127.0.0.1:8082;

[2026-05-27T01:43:41Z] proxy_http_version 1.1;

[2026-05-27T01:43:41Z] proxy_set_header Host $host;

[2026-05-27T01:43:41Z] proxy_set_header X-Real-IP $remote_addr;

[2026-05-27T01:43:41Z] proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

[2026-05-27T01:43:41Z] proxy_set_header X-Forwarded-Proto $scheme;

[2026-05-27T01:43:41Z]

[2026-05-27T01:43:41Z] }

[2026-05-27T01:43:41Z]

[2026-05-27T01:43:41Z] server_name _;

[2026-05-27T01:43:41Z]

[2026-05-27T01:43:41Z] }

[2026-05-27T01:43:41Z] [2026-05-27T01:43:41Z INFO oak_webrtc] Waiting for WebRTC connection...

[2026-05-27T01:43:41Z] [2026-05-27 01:43:41.223] [depthai] [warning] USB protocol not available - If running in a container, make sure that the following is set: "-v /dev/bus/usb:/dev/bus/usb --device-cgroup-rule='c 189:* rmw'"

[2026-05-27T01:43:41Z] [2026-05-27T01:43:41Z INFO lib-webrtc] Connection to signaling server established url=wss://signal.cloud.luxonis.com/agent/

[2026-05-27T01:43:41Z] 2026/05/27 01:43:41 [error] 37#37: *2 connect() failed (111: Connection refused) while connecting to upstream, client: ::1, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8082/", host: "localhost:9000", referrer: "http://localhost:9000/"

[2026-05-27T01:43:42Z] 2026/05/27 01:43:42 [error] 37#37: *2 connect() failed (111: Connection refused) while connecting to upstream, client: ::1, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8082/", host: "localhost:9000", referrer: "http://localhost:9000/"

[2026-05-27T01:43:43Z] 2026/05/27 01:43:43 [error] 37#37: *2 connect() failed (111: Connection refused) while connecting to upstream, client: ::1, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8082/", host: "localhost:9000", referrer: "http://localhost:9000/"

[2026-05-27T01:43:44Z] 2026/05/27 01:43:44 [error] 37#37: *2 connect() failed (111: Connection refused) while connecting to upstream, client: ::1, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8082/", host: "localhost:9000", referrer: "http://localhost:9000/"

[2026-05-27T01:43:45Z] 2026/05/27 01:43:45 [error] 37#37: *2 connect() failed (111: Connection refused) while connecting to upstream, client: ::1, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8082/", host: "localhost:9000", referrer: "http://localhost:9000/"

[2026-05-27T01:43:45Z] To connect to the OAK visualizer use the following:

[2026-05-27T01:43:45Z] If you are opening the visualizer locally, open http://localhost:8082 in your browser

[2026-05-27T01:43:45Z] If you are using peripheral mode and viewing from another machine, open http://10.10.3.7:8082 in your browser

[2026-05-27T01:43:45Z] If you are using oakapp base image, open https://10.10.3.7:9000 in your browser

Open frontend here: http://10.10.3.7:9000/

[2026-05-27T01:43:51Z] CONNECTION - Agent request succeeded

[2026-05-27T01:43:51Z] CONNECTION - To connect DepthAI Viewer locally, open: https://10.10.3.7:9000

[2026-05-27T01:43:56Z] 2026/05/27 01:43:56 [crit] 38#38: *43 mkdir() "/var/lib/nginx/proxy/1" failed (28: No space left on device) while reading upstream, client: 10.10.3.98, server: _, request: "GET /assets/index-O9sBLzcc.css HTTP/1.1", upstream: "http://127.0.0.1:8082/assets/index-O9sBLzcc.css", host: "10.10.3.7:9000", referrer: "https://10.10.3.7:9000/"

The app appears to be running but when going to the viewer at 10.10.3.7:9000 it still shows the same blank white page along with the error in logs indicating device is running out of space. At this point the only option left seems to be a master device reset.

OskarSonc

oakleydoakley - thanks for the detailed update. This now looks like the device is running out of local storage, and the blank white page is likely just a symptom of that. The nginx errors show it cannot create temporary proxy files, so the frontend assets are not being served correctly.

There are two known causes that fit what you are seeing:

leftover app/container storage not being fully reclaimed after app deletion
a Luxonis OS issue in older releases where persistent logs can grow until the disk fills up

Could you please share the output of oakctl device info so we can confirm the OS version?
If you can access the device shell via SSH or ADB, please also try clearing the logs with:

rm -f /overlay/persistent-logs/*
rm -f /var/volatile/log/*

If the device is already too full to recover cleanly after that, a factory reset or full OS flash may be the quickest path back to a working state.

Best,
Oskar

oakleydoakley

@OskarSonc OS version on both cameras is 1.31.2.

Before your reply, I investigated further by sshing into the camera to look at the storage usage.

~ # df -a
Filesystem 1K-blocks Used Available Use% Mounted on
proc 0 0 0 0% /proc
sysfs 0 0 0 0% /sys
devtmpfs 4096 0 4096 0% /dev
tmpfs 3687372 0 3687372 0% /dev/shm
selinuxfs 0 0 0 0% /sys/fs/selinux
devpts 0 0 0 0% /dev/pts
tmpfs 1474952 93500 1381452 6% /run
tmpfs 4096 0 4096 0% /sys/fs/cgroup
cgroup2 0 0 0 0% /sys/fs/cgroup/unified
cgroup 0 0 0 0% /sys/fs/cgroup/systemd
bpf 0 0 0 0% /sys/fs/bpf
cgroup 0 0 0 0% /sys/fs/cgroup/cpu,cpuacct
cgroup 0 0 0 0% /sys/fs/cgroup/freezer
cgroup 0 0 0 0% /sys/fs/cgroup/devices
cgroup 0 0 0 0% /sys/fs/cgroup/net_cls,net_prio
cgroup 0 0 0 0% /sys/fs/cgroup/pids
cgroup 0 0 0 0% /sys/fs/cgroup/blkio
cgroup 0 0 0 0% /sys/fs/cgroup/cpuset
cgroup 0 0 0 0% /sys/fs/cgroup/memory
mqueue 0 0 0 0% /dev/mqueue
debugfs 0 0 0 0% /sys/kernel/debug
tracefs 0 0 0 0% /sys/kernel/tracing
configfs 0 0 0 0% /sys/kernel/config
fusectl 0 0 0 0% /sys/fs/fuse/connections
/dev/sda7 5068448 4370008 682056 86% /
mtp 0 0 0 0% /dev/usb-ffs/mtp
adb 0 0 0 0% /dev/usb-ffs/adb
gate 0 0 0 0% /dev/usb-ffs/gate
device 0 0 0 0% /dev/usb-ffs/device
diag 0 0 0 0% /dev/ffs-diag
tmpfs 80761644 78018072 0 100% /etc/machine-id
tmpfs 3687372 20056 3667316 1% /tmp
/dev/sde33 65488 1360 64128 2% /bt_firmware
/dev/sde37 60372 42672 16392 72% /dsp
/dev/sde32 327568 198432 129136 61% /firmware
/dev/sda2 2916 40 2736 1% /persist
binder 0 0 0 0% /dev/binderfs
tmpfs 3687372 36 3687336 0% /var/volatile
overlay 3687372 36 3687336 0% /var/cache
overlay 3687372 36 3687336 0% /srv
overlay 3687372 36 3687336 0% /var/lib
overlay 3687372 36 3687336 0% /var/spool
/dev/sda11 80761644 78018072 0 100% /overlay
overlay 80761644 78018072 0 100% /cache
overlay 80761644 78018072 0 100% /etc
overlay 80761644 78018072 0 100% /data
overlay 80761644 78018072 0 100% /home
/dev/sda7 5068448 4370008 682056 86% /usr/share/vulkan/icd.d/lvp_icd.aarch64.json
/dev/sda7 5068448 4370008 682056 86% /usr/share/vulkan/icd.d/lvp_icd.aarch64.json
/dev/sda7 5068448 4370008 682056 86% /usr/local/share/vulkan/icd.d/adrenovk.json
oak-agent-image-fa035994508906e6a0a6d4d74541c43469a67ed0ef0a7a96aa52f4ea25deb01b
80761644 78018072 0 100% /overlay/oak-agent/containers/images/fa035994508906e6a0a6d4d74541c43469a67ed0ef0a7a96aa52f4ea25deb01b/merged
oak-agent-image-fee363c5190eaa6053079c17216db3ea4cc7253027ca08296f3aaa4d4b69028f
80761644 78018072 0 100% /overlay/oak-agent/containers/images/fee363c5190eaa6053079c17216db3ea4cc7253027ca08296f3aaa4d4b69028f/merged
oak-agent-image-fb4bd1bc9ec70454614ebcdb49b21a54513f16018835a4e2ab2d5fc67d9b3bbb
80761644 78018072 0 100% /overlay/oak-agent/containers/images/fb4bd1bc9ec70454614ebcdb49b21a54513f16018835a4e2ab2d5fc67d9b3bbb/merged
oak-agent-container-fb44b3d1-f0f1-4599-8157-77760f4eb05f
80761644 78018072 0 100% /overlay/oak-agent/containers/containers/fb44b3d1-f0f1-4599-8157-77760f4eb05f/merged
oak-agent-container-1b9aed79-4efb-4f7b-a0a7-f5aa9f1bdff9
80761644 78018072 0 100% /overlay/oak-agent/containers/containers/1b9aed79-4efb-4f7b-a0a7-f5aa9f1bdff9/merged
tracefs 0 0 0 0% /sys/kernel/debug/tracing
~ #

It's clear that /dev/sda11 is where all the containers for the apps are stored and is completely full at 100%. There are 5 containers shown but no apps installed. This confirms my suspicion that oakctl is not removing unused containers even after an application has been uninstalled.

I manually unmounted /dev/sda11 and deleted the containers to free up the space:
~ # df -h /overlay
Filesystem Size Used Available Use% Mounted on
/dev/sda11 77.0G 2.3G 70.7G 3% /overlay

Then was successfully able to install the app and I'm able to access the liveview on 10.10.3.6:9000 again.

So this confirms this is a storage issue. Luxonis needs to resolve this and prevent this from happening. We shouldn't have to manage the storage manually on our own.

OskarSonc

oakleydoakley - thanks for diging into this! Reported to dev team and they already have this fix and will be released in new oakctl.
Thanks for waiting