DeepLabV3 on Luxonis Oak-1 camera

AlejandroDiaz

Hello,

I need to build a segmentation application and I notice that on the documentation it states that DeepLabV3 is tested for the camera devices. I have a Oak-1 W camera connected to a raspberry pi and I'm trying to load the default deeplab_v3_mnv2_513x513 model to my luxonis device. However I'm getting the following error message

Does anybody have experience deploying DeepLab on Luxonis cameras?

jakaskerl

AlejandroDiaz
Segfault likely because qt or some other app ins not install or there are environment issues. Show output of pip list, make sure you have x11 forwarding enabled, and paste the code.

Thanks,
Jaka

AlejandroDiaz

Here is the list of the packages installed

Package Version

---------------------------------- ----------

arandr 0.1.11

asgiref 3.6.0

astroid 2.14.2

asttokens 2.2.1

av 10.0.0

Babel 2.10.3

beautifulsoup4 4.11.2

blinker 1.5

blobconverter 1.4.2

boto3 1.28.63

botocore 1.31.63

certifi 2022.9.24

chardet 5.1.0

charset-normalizer 2.0.12

click 8.1.3

colorama 0.4.6

colorzero 2.0

cryptography 38.0.4

cupshelpers 1.0

dbus-python 1.3.2

depthai 2.22.0.0

depthai-pipeline-graph 0.0.5

depthai-sdk 1.12.1

dill 0.3.6

distinctipy 1.2.3

distro 1.8.0

docutils 0.19

ffmpy3 0.2.4

Flask 2.2.2

gpiozero 2.0

html5lib 1.1

idna 3.3

importlib-metadata 4.12.0

isort 5.6.4

itsdangerous 2.1.2

jedi 0.18.2

Jinja2 3.1.2

jmespath 1.0.1

lazy-object-proxy 1.9.0

lgpio 0.2.2.0

libevdev 0.5

logilab-common 1.9.8

lxml 4.9.2

lz4 4.3.2

MarkupSafe 2.1.2

marshmallow 3.17.0

mcap 1.1.1

mcap-ros1-support 0.0.8

mccabe 0.7.0

more-itertools 8.10.0

mypy 1.0.1

mypy-extensions 0.4.3

numpy 1.26.1

oauthlib 3.2.2

olefile 0.46

opencv-contrib-python 4.5.5.62

packaging 23.2

parso 0.8.3

pexpect 4.8.0

pgzero 1.2

picamera2 0.3.12

pidng 4.0.9

piexif 1.1.3

pigpio 1.78

Pillow 9.4.0

pip 23.0.1

platformdirs 2.6.0

psutil 5.9.4

ptyprocess 0.7.0

pycairo 1.20.1

pycups 2.0.1

pygame 2.1.2

Pygments 2.14.0

PyGObject 3.42.2

pyinotify 0.9.6

PyJWT 2.6.0

pylint 2.16.2

PyOpenGL 3.1.6

pyOpenSSL 23.0.0

PyQt5 5.15.9

PyQt5-sip 12.11.1

pyserial 3.5

pysmbc 1.0.23

python-apt 2.6.0

python-dateutil 2.8.2

python-dotenv 0.21.0

python-prctl 1.8.1

pytube 15.0.0

PyTurboJPEG 1.6.4

pytz 2022.7.1

pyudev 0.24.0

pyusb 1.2.1

PyYAML 6.0.1

Qt.py 1.3.8

reportlab 3.6.12

requests 2.31.0

requests-oauthlib 1.3.0

responses 0.18.0

roman 3.3

rosbags 0.9.11

RPi.GPIO 0.7.1a4

RTIMULib 7.2.1

ruamel.yaml 0.17.35

ruamel.yaml.clib 0.2.8

s3transfer 0.7.0

scipy 1.11.3

Send2Trash 1.8.1b0

sense-hat 2.6.0

sentry-sdk 1.21.0

setuptools 66.1.1

simplejpeg 1.6.6

simplejson 3.18.3

six 1.16.0

smbus2 0.4.2

soupsieve 2.3.2

speedtest-cli 2.1.3

spidev 3.5

ssh-import-id 5.10

thonny 4.1.3

toml 0.10.2

tomlkit 0.11.7

twython 3.8.2

types-aiofiles 22.1

types-annoy 1.17

types-appdirs 1.4

types-aws-xray-sdk 2.10

types-babel 2.11

types-backports.ssl-match-hostname 3.7

types-beautifulsoup4 4.11

types-bleach 5.0

types-boto 2.49

types-braintree 4.17

types-cachetools 5.2

types-caldav 0.10

types-certifi 2021.10.8

types-cffi 1.15

types-chardet 5.0

types-chevron 0.14

types-click-spinner 0.1

types-colorama 0.4

types-commonmark 0.9

types-console-menu 0.7

types-contextvars 2.4

types-croniter 1.3

types-cryptography 3.3

types-D3DShot 0.1

types-dateparser 1.1

types-DateTimeRange 1.2

types-decorator 5.1

types-Deprecated 1.2

types-dj-database-url 1.0

types-docopt 0.6

types-docutils 0.19

types-editdistance 0.6

types-emoji 2.1

types-entrypoints 0.4

types-first 2.0

types-flake8-2020 1.7

types-flake8-bugbear 22.10.27

types-flake8-builtins 2.0

types-flake8-docstrings 1.6

types-flake8-plugin-utils 1.3

types-flake8-rst-docstrings 0.2

types-flake8-simplify 0.19

types-flake8-typing-imports 1.14

types-Flask-Cors 3.0

types-Flask-SQLAlchemy 2.5

types-fpdf2 2.5

types-gdb 12.1

types-google-cloud-ndb 1.11

types-hdbcli 2.14

types-html5lib 1.1

types-httplib2 0.21

types-humanfriendly 10.0

types-invoke 1.7

types-JACK-Client 0.5

types-jmespath 1.0

types-jsonschema 4.17

types-keyboard 0.13

types-ldap3 2.9

types-Markdown 3.4

types-mock 4.0

types-mypy-extensions 0.4

types-mysqlclient 2.1

types-oauthlib 3.2

types-openpyxl 3.0

types-opentracing 2.4

types-paho-mqtt 1.6

types-paramiko 2.11

types-parsimonious 0.10

types-passlib 1.7

types-passpy 1.0

types-peewee 3.15

types-pep8-naming 0.13

types-Pillow 9.3

types-playsound 1.3

types-polib 1.1

types-prettytable 3.4

types-protobuf 3.20

types-psutil 5.9

types-psycopg2 2.9

types-pyaudio 0.2

types-PyAutoGUI 0.9

types-pycurl 7.45

types-pyfarmhash 0.3

types-pyflakes 2.5

types-Pygments 2.13

types-pyinstaller 5.6

types-PyMySQL 1.0

types-pynput 1.7

types-pyOpenSSL 22.1

types-pyRFC3339 1.1

types-PyScreeze 0.1

types-pysftp 0.2

types-pyside2 5.15.2.1.6

types-pytest-lazy-fixture 0.6

types-python-crontab 2.6

types-python-dateutil 2.8

types-python-gflags 3.1

types-python-jose 3.3

types-python-nmap 0.7

types-python-slugify 6.1

types-pytz 2022.6

types-pyvmomi 7.0

types-pywin32 304

types-PyYAML 6.0

types-redis 4.3

types-regex 2022.10.31

types-requests 2.28

types-retry 0.9

types-Send2Trash 1.8

types-setuptools 65.5

types-simplejson 3.17

types-singledispatch 3.7

types-six 1.16

types-slumber 0.7

types-SQLAlchemy 1.4.43

types-stdlib-list 0.8

types-stripe 3.5

types-tabulate 0.9

types-termcolor 1.1

types-toml 0.10

types-toposort 1.7

types-tqdm 4.64

types-tree-sitter 0.20

types-tree-sitter-languages 1.5

types-ttkthemes 3.2

types-typed-ast 1.5

types-tzlocal 4.2

types-ujson 5.5

types-urllib3 1.26

types-vobject 0.9

types-waitress 2.1

types-whatthepatch 1.0

types-xmltodict 0.13

types-xxhash 3.0

types-zxcvbn 4.4

typing_extensions 4.4.0

urllib3 1.26.12

v4l2-python3 0.3.2

webencodings 0.5.1

Werkzeug 2.2.2

wheel 0.38.4

wrapt 1.14.1

xmltodict 0.13.0

zipp 1.0.0

zstandard 0.21.0

AlejandroDiaz

import depthai as dai

pipeline = dai.Pipeline()

nn_path = blobconverter.from_zoo(name="deeplab_v3_mnv2_256x256", zoo_type="depthai", shaves=6)

model_nn = pipeline.create(dai.node.NeuralNetwork)

model_nn.setBlobPath(nn_path)

model_nn.setNumInferenceThreads(2)

model_nn.input.setBlocking(False)

cam_rgb = pipeline.create(dai.node.ColorCamera)

cam_rgb.setPreviewSize(256, 256)

cam_rgb.setInterleaved(False)

cam_rgb.preview.link(model_nn.input)

xout_nn = pipeline.create(dai.node.XLinkOut)

xout_nn.setStreamName("nn")

model_nn.out.link(xout_nn.input)

# Connect to device

with dai.Device(pipeline) as device:

q_nn = device.getOutputQueue(name="nn", maxSize=4, blocking=False)

while True: in_nn = q_nn.get()

jakaskerl

AlejandroDiaz
Ok thanks, can you try with a different model? And make sure depthai is latest (2.28) and not 2.22.

Thanks
Jaka

AlejandroDiaz

It doesn't work with other model, I tried yolov8n. Also when I tried to updated depthAI using pip I get the following message

pip install -U depthai

error: externally-managed-environment

× This environment is externally managed

╰─> To install Python packages system-wide, try apt install

python3-xyz, where xyz is the package you are trying to

install.



If you wish to install a non-Debian-packaged Python package,

create a virtual environment using python3 -m venv path/to/venv.

Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make

sure you have python3-full installed.



If you wish to install a non-Debian packaged Python application,

it may be easiest to use pipx install xyz, which will manage a

virtual environment for you. Make sure you have pipx installed.



See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.

hint: See PEP 668 for the detailed specification.

AlejandroDiaz

I'm using an RPI 4 as host and it has the one of the images that is available on your documentation drive.

jakaskerl

AlejandroDiaz
if you have OS image from drive then you should have environment installed already. Check under luxonis or desktop or depthai-python. I think we named it .env. Activate it and try again.

Thanks,
Jaka

AlejandroDiaz

I couldn't find the external environment. Therefore, I decide to try using the Docker image using as my base image
luxonis/depthai-library:latest

In that case I'm getting the following message

I ran the script available here and it uploads the bootloader from 0.0.22 to 0.0.26 but then it's not moving to 0.0.28

jakaskerl

AlejandroDiaz
I'd rather you then create your own envirnoment, activate it, then run python examples/install_requirements.py

AlejandroDiaz

I also did that experiment, created my custom environment. Update depthai to 2.28, flash the bootloader to 0.0.28. But the result is the same. What I noticed is that when I create a pipeline just for viewing the image, it works. But when I add the NueralNetwork Node to my pipeline is when it crash with the error I reported.

jakaskerl

AlejandroDiaz
Ok. Can you add a short MRE for both the script that works and the one that doesn't. Make the changes as small as possible. It should not happen.

My thoughts:

increases power draw causes the crash
some operation on host causes the crash

Thanks,
Jaka

AlejandroDiaz

This s the script that works okay, when I add the NN node to the pipeline is when it's crashing. Could it be that the PoE switch that I'm using doesn't support 1000 Mbps? It only supports 10/100 Mbps

import cv2

import sys

import depthai as dai

import blobconverter

from typing import List

# Check if the MX ID is provided as an argument

if len(sys.argv) < 2:

print("Usage: python capture_image.py <device_mxid>")

sys.exit(1)

# Get the MX ID from the command-line argument

device_mxid = sys.argv[1]

# Function to create a pipeline to capture RGB images

def create_pipeline():

# Create a pipeline

pipeline = dai.Pipeline()

## When I uncomment this lines in my pipeline it causes the crash

# nn_path = blobconverter.from_zoo(name="mobilenet-v3-small-1.0-224-tf", shaves=6)

# model_nn = pipeline.create(dai.node.NeuralNetwork)

# model_nn.setBlobPath(nn_path)  # Provide the path to your custom model

# Define a source - color camera

cam_rgb = pipeline.create(dai.node.ColorCamera)

cam_rgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)

# Create an output stream

xout_rgb = pipeline.create(dai.node.XLinkOut)

xout_rgb.setStreamName("rgb")

cam_rgb.video.link(xout_rgb.input)

return pipeline

# Discover devices and select the one with the specified MX ID

infos: List[dai.DeviceInfo] = dai.DeviceBootloader.getAllAvailableDevices()

if len(infos)>0:

for info in infos:

    if info.getMxId() == device_mxid:

        print(f"Found device with MX ID: {device_mxid}")

        # Create a pipeline

        pipeline = create_pipeline()

        

        # Connect to the device

        with dai.Device(pipeline, info) as device:

            # Output queue for RGB frames

            q_rgb = device.getOutputQueue(name="rgb", maxSize=4, blocking=True)

            # Get the RGB frame from the output queue (this captures one frame)

            in_rgb = q_rgb.get()  # Blocking call, waits for a frame

            # Convert the frame to a format OpenCV can use (BGR)

            frame = in_rgb.getCvFrame()

            # Save the frame as an image

            image_path = f"luxonis_image_{device_mxid}.png"

            cv2.imwrite(image_path, frame)

            print(f"Image saved at: {image_path}")

    else:

        print(f"No devices found with MX ID: {device_mxid}")

else:

print("No devices found!!")

jakaskerl

AlejandroDiaz
Hm, strange. Can you paste the script with NN as well? I'm interested to see how you linked the nodes together and retrieved the results.

Thanks,
Jaka

AlejandroDiaz

It was a power error, changing the switch with one with more power fixed the issue.