Illegal instruction (core dumped) when running depthai demo on jetson xavier nx

rymd80

I'm running the depthai install and demo in a python virtualenv on a jetson xavier nx which yields a core dump. My other python virtualenv on the jetson that runs tensorflow runs with no core dump.

Any clue on resolving the core dump would be much appreciated.

Background
The jetson xavier nx is running Ubuntu 18.04.

I commented out the open3d==0.10.0.0 line in the requirement file to get the install_requirments.py to run successfully.

I installed opencv 4.5.1 using a script from https://github.com/mdegans/nano_build_opencv.

I followed the depthai install dependencies instructions before running the install_requirments.py.

Terminal Output showing depthai successful install and running demo (with core dump), and then import of depthai (loads successfully), cv2, numpy (both generate a core dump).

(depthai) user@jetson-desktop:_{/dev/depthai$} python3 install_requirements.py
Requirement already satisfied: pip in ./depthai/lib/python3.6/site-packages (21.0.1)
WARNING: Skipping depthai as it is not installed.
Requirement already satisfied: numpy==1.19.5 in ./depthai/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (1.19.5)
Requirement already satisfied: opencv-python==4.5.1.48 in ./depthai/lib/python3.6/site-packages (from -r requirements.txt (line 2)) (4.5.1.48)
Requirement already satisfied: requests==2.24.0 in ./depthai/lib/python3.6/site-packages (from -r requirements.txt (line 3)) (2.24.0)
Requirement already satisfied: argcomplete==1.12.1 in ./depthai/lib/python3.6/site-packages (from -r requirements.txt (line 4)) (1.12.1)
Requirement already satisfied: pyyaml==5.3.1 in ./depthai/lib/python3.6/site-packages (from -r requirements.txt (line 5)) (5.3.1)
Collecting depthai==0.4.1.1
Using cached depthai-0.4.1.1-cp36-cp36m-linux_aarch64.whl
Requirement already satisfied: importlib-metadata<3,>=0.23 in ./depthai/lib/python3.6/site-packages (from argcomplete==1.12.1->-r requirements.txt (line 4)) (2.1.1)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in ./depthai/lib/python3.6/site-packages (from requests==2.24.0->-r requirements.txt (line 3)) (1.25.11)
Requirement already satisfied: idna<3,>=2.5 in ./depthai/lib/python3.6/site-packages (from requests==2.24.0->-r requirements.txt (line 3)) (2.10)
Requirement already satisfied: chardet<4,>=3.0.2 in ./depthai/lib/python3.6/site-packages (from requests==2.24.0->-r requirements.txt (line 3)) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in ./depthai/lib/python3.6/site-packages (from requests==2.24.0->-r requirements.txt (line 3)) (2020.12.5)
Requirement already satisfied: zipp>=0.5 in ./depthai/lib/python3.6/site-packages (from importlib-metadata<3,>=0.23->argcomplete==1.12.1->-r requirements.txt (line 4)) (3.4.0)
Installing collected packages: depthai
Successfully installed depthai-0.4.1.1

(depthai) user@jetson-desktop:_{/dev/depthai$} python3 depthai_demo.py
Illegal instruction (core dumped)

(depthai) user@jetson-desktop:_{/dev/depthai$} python3
Python 3.6.9 (default, Oct 8 2020, 12:12:24)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import depthai
quit()

(depthai) user@jetson-desktop:_{/dev/depthai$} python3
Python 3.6.9 (default, Oct 8 2020, 12:12:24)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import cv2
Illegal instruction (core dumped)

(depthai) user@jetson-desktop:_{/dev/depthai$} python3
Python 3.6.9 (default, Oct 8 2020, 12:12:24)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import numpy
Illegal instruction (core dumped)

rymd80

Seems like I at least partially found the solution, at least for the first error.

I realized that the architecture aarch64 is the key.

opencv-python has a download for aarch64 in 3.4.13.47, and then not again until 4.5.1.

I backed down to opencv to 3.4.13 and opencv-python to 3.4.13.47 and the core dump goes away.

I haven't figured why opencv 4.5.1 doesn't work.

However, now I'm getting an error when downloading the model (I get the same error when attempting to run calibrate):

Note: when I edit depthai/resources/nn/openpose/model.yml, I see references to "human_pose," but nothing for "openpose."

ask_type: human_pose_estimation
files:

name: FP32/human-pose-estimation-0001.xml

python3 depthai_demo.py
No calibration file. Using Calibration Defaults.
Using depthai module from: /home/user/Documents/workspace/projects/face-mask-detector/face-mask-detector/lib/python3.6/site-packages/depthai.cpython-36m-aarch64-linux-gnu.so
Depthai version installed: 0.4.1.1+dev
Depthai development version found, skipping check.
Compiling model for 14 shaves, 14 cmx_slices and 1 NN_engines
model_compilation_target: cloud
In config "/home/user/Documents/workspace/depthai/resources/nn/openpose/model.yml":
In model "openpose":
No XML file for precision "FP32"
Traceback (most recent call last):
File "depthai_demo.py", line 415, in <module>
dai.startLoop()
File "depthai_demo.py", line 45, in startLoop
configMan = DepthConfigManager(args)
File "/home/user/Documents/workspace/depthai/depthai_helpers/config_manager.py", line 26, in init
self.generateJsonConfig()
File "/home/user/Documents/workspace/depthai/depthai_helpers/config_manager.py", line 192, in generateJsonConfig
blobMan = BlobManager(self.args, self.calc_dist_to_bb, shave_nr, cmx_slices, NCE_nr)
File "/home/user/Documents/workspace/depthai/depthai_helpers/config_manager.py", line 377, in init
self.blob_file = self.compileBlob(self.args['cnn_model'], self.args['model_compilation_target'])
File "/home/user/Documents/workspace/depthai/depthai_helpers/config_manager.py", line 431, in compileBlob
ret = download_and_compile_NN_model(nn_model, model_zoo_folder, shave_nr_opt, cmx_slices_opt, NCE_nr, outblob_file, model_compilation_target)
File "/home/user/Documents/workspace/depthai/model_compiler/model_compiler.py", line 154, in download_and_compile_NN_model
download_location = download_model(model, model_zoo_folder)
File "/home/user/Documents/workspace/depthai/model_compiler/model_compiler.py", line 32, in download_model
raise RuntimeError("Model downloader failed!")
RuntimeError: Model downloader failed!

Brandon

rymd80 Sorry about the delay here.

So as an update, there is a related PR here for the build for Jetson (Nano, at least, but I think Xavier should work as well).

https://github.com/luxonis/depthai/pull/329

On the model downloader failing, is the Xavier connected to the internet? And can it access http://luxonis.com:8080/ ?

I'll ask Szabi if he has any ideas on the model downloader as well (he wrote it).

Thanks,
Brandon

GergelySzabolcs

rymd80
Hello.
Can you try on a clean repo?
I think you edited the model.yml. It shouldn't be edited, unless custom model is added.

Openpose is the same as human-pose-estimation-0001, openpose folder is a symlink for backward compatibility.
Just run: python3 depthai_demo.py -cnn openpose

rymd80

GergelySzabolcs

I didn't edit, but a git status shows:

On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
(use "git add/rm <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)

modified:   requirements-optional.txt
modified:   requirements.txt
deleted:    resources/nn/openpose
deleted:    resources/nn/openpose2

Untracked files:
(use "git add <file>..." to include in what will be committed)

install_dependencies.sh

No idea how those deletes happened... I'm not normally into randomly editing or deleting files that aren't mine, but I've done stranger things...

Then I did

git reset --hard HEAD
git fetch
git checkout pi_zero_wheel
python3 install_requirements.py
python3 depthai_demo.py -rgbf 5

and demo ran successfully.

(Oh I had to comment out open3d in requirements-optional.txt)

Thanks for the help.

GergelySzabolcs

rymd80
Good to know, thanks.
The failure of open3d installation is not an error, since it's only optional. (Not supported on all platforms)

CactusJackFX

I know this is a late to the party, but I built OpenCV for my Jetson Xavier AGX, and I too got the error when I tried to use CV2. Here is the information that I found that worked really well to resolve that issue:

CV2 Illegal Instructions Resolution:
add export OPENBLAS_CORETYPE=ARMV8 to your .bashrc should fix it.

Brandon

Thanks CactusJackFX for sharing!

Letty

CactusJackFX thank you, your solution worked for me too on a Jetson Xavier NX!

@Brandon is the solution for the build on Jetson valid only for Gen 1 at the moment? Do you know if there is there a way to build DepthAI Gen 2 on a Jetson?

Brandon

Letty So @themarpe actually knows on this. I risk being wrong on the Jetson build for Gen2. I've only personally built Gen1 on Jetson.

And I think @themarpe is actually adding Jetson (aarch64) series to our CI/CD next week so that these will be pre-built along with Ubuntu-AM64/Raspberry Pi OS/macOS/Windows. We had been meaning to do this earlier, but just a bit behind.

So potentially Martin may reply with build instructions and that it is now prebuilt. :-).

Best,
Brandon

Letty

Brandon Thanks a lot Brandon!

Brandon

I meant to circle back @Letty that this is actually already ready (as of 21 hours ago):

https://github.com/luxonis/depthai-python/pull/158

Letty

Brandon Thanks a lot!