OAK-D-Lite with Yolov7 On a Jetson Nano

jakaskerl

Hi SeanWorcester
Make sure to also go through all the packages inside install_dependencies.sh. Seems like you are missing more than you said above. udev seems to be one of them.

You should have `/etc/udev/rules.d/ folder present on your system. Then you will be able to write the rules to .rules file.

udevadm is used to reload and apply the changes without the need to restart your PC.

Thanks,
Jaka

SeanWorcester

Hello jakaskerl

I did go though every package on install_dependencies.sh the only two i couldn't install were libdc1394-dev and libjasper-dev. I did a check line by line and everything else is accounted for and installed.

I checked a working individual DepthAI install on another SDcard and I saw those same two packages were not installed there.

Could there be something else I am missing?

Thanks,

SeanWorcester

Hello jakaskerl

I compared the .rules folders of a standard install of DepthAI with the modified install that I put in my Yolov7 environment and honestly I don't see a difference.

Do you know what rule I need to add to get the OAK-D-Lite working had and where?

Thank you,

jakaskerl

Hi SeanWorcester
If you are concerned about your current environment, you can make a backup: pip freeze >> backup.txt. That way you can run pip install -r backup.txt and recreate your env. I'd suggest running the install_dependencies as it doesn't change your environment, and then run install_requirements.py as well. If if breaks your current working environment, you can just restore it from the backup.

I will check more on monday.

Thanks,
Jaka

SeanWorcester

Hello jakaskerl

I followed your suggestion and it works!! I can run the DepthAI and Yolov7 in the same environment and they don't seem to clash. I am able to run rgb_preview.py and the Yolov7 detect.py with no issues. Now I hope to use the OAK-D-Lite rgb camera for Yolov7 and reference my detection with the StereoDepth camera. Still need to fix this part out.

Thank you for all the help you have given me.

SeanWorcester

Hello @jakaskerl

I have a follow on questions along the same topic.

Can the the OAK-D-Lite run Yolov7 on-board?

Will the instructions at,

luxonis/depthai-ml-trainingblob/master/colab-notebooks/YoloV7_training.ipynb

Work with a OAK-D-Lite?

Thank you,

jakaskerl

Hi @SeanWorcester
They will; ...with any oak device that has an RGB camera for that matter.

Thanks,
Jaka

SeanWorcester

Hi @jakaskerl

Thank you for the reply!
I read through the documentation for making a custom Yolov7 tiny model, and I have a couple of questions.

First, after I create the Yolov7 tiny model and convert it to the BLOB format do call my model like this in Python?

self._spatialDetectionNetwork.setBlobPath(BLOB_PATH)

Where BLOB_PATH is the location of the new model on my Jetson Nano.

Second, do I need to change anything on the OAK-D-Lite camera itself or run this new model?

Thank You,

jakaskerl

Hi @SeanWorcester
That's how you specify the path so the node knows which model to use, yes. Since it's a v7 model which iirc does not have anchors/masks, you typically don't need to change anything other than the input size (if different) and maybe the confidence threshold.

Thanks,
Jaka

SeanWorcester

Hello jakaskerl

I created a Yolov7-tiny model and made sure it worked with Yolov7 before converting it to a BLOB file. Then I modified the spatial_tiny_yolo.py example script. and called the new Yolov7-tiny BLOP file. I also changed the self._camRgb.setPreviewSize() to 640x480

when I ran the modified example I got,

I changed self._camRgb.setPreviewSize() from 640x480 to 640x640, and ran it again.

my error changed to,

Have you seen this issue before or know what else I need to change?

Thank you for your help!

jakaskerl

SeanWorcester

SeanWorcester I changed self._camRgb.setPreviewSize() from 640x480 to 640x640, and ran it again.

But you have set your model to run on 640x480? Looks like a wrong blob path specified then. cc @Matija

Thanks,
Jaka

Matija

It looks like the input shape expected by the model is 640x640 while model expects 640x480.

I suggest you use our yolov7_training notebook you shared above, export it using tools.luxonis.com, and then use luxonis/depthai-experimentsblob/master/gen2-yolo/device-decoding/main_api.py to run the network. This will verify that the export is correct and everything should work

SeanWorcester

Hello Matija and jakaskerl

Did I create the model wrong? I looked at the the Notebook above and mostly followed the steps except in a few areas.

I resized all my images to 640x480 before I prepared all my training and validation data.

when ran train.py I used the following arguments.

python train.py --batch 16 --cfg cfg/training/custom_yolov7-tiny.yaml --epochs 300 --data /content/yolov7/data/custom.yaml --weights 'yolov7-tiny.pt' --device 0

Which I see defaults the image data to 640x640.

Could this be my problem? Do I need to resize my images to 640x640 before preparing all my data?

Thank you for all the help!

jakaskerl

Hi @SeanWorcester
I don't see an explicit flag to set the input size like done in the notebook:

!python train.py --epochs 2 --workers 8 --device 0 --batch-size 32 --data data/voc.yaml --img 640 640 --cfg cfg/training/yolov7_voc-tiny.yaml --weights 'yolov7-tiny.pt' --hyp data/hyp.scratch.tiny.yaml

Did you miss changing the size somewere; including https://tools.luxonis.com?

Thanks,
Jaka

SeanWorcester

Hello jakaskerl

Correct I missed the --img flag, but I see that the default is 640x640. I have a couple of questions before I build my model again.

Should I resize my images to 640x640 before generating labels, or should I leave them at 3000x4000 before running train.py?

When I entered the best.py into the BLOB converter tool I specified the shape at 640. Should I do 640x640 this time?

Thank you so much for your help!

jakaskerl

Hi @SeanWorcester
I'm not sure this is needed, but if 3000x4000 don't give you any errors, it's likely ok.

SeanWorcester When I entered the best.py into the BLOB converter tool I specified the shape at 640. Should I do 640x640 this time?

Either 640 or 640 640 is fine. But isn't the model 640 480?

Thanks,
Jaka

SeanWorcester

Hi jakaskerl

I am thinking I will try rebuilding my model with 3000x4000 images and and set the -img flag to 640x640. Then set BLOB converter to 640x640.

Then in the Spartal code would I set the preview to 640x640?

Thank you!

SeanWorcester

Hi @jakaskerl

I didn't set the -img flag so I believe the default is 640x640. I used images that were sized to 640x480 during training.

I am wondering what would happen if I set the training.py -img to 640x480, and created a new model; then set set preview() to 640x480.

Do you think this would work?

Thanks!

Matija

SeanWorcester

The images should keep the aspect ratio when you train them. When you are resizing them on OAK you should keep the aspect ratio too. If you use the preview feature of the RGB camera node, make sure to use setPreviewKeepAspectRatio(True) on the node.

If you will keep square input shape, then preview shape must be set to 640 x 640 or similar, and some of the FOV will be cropped. If you have 3000x4000 I would recommend setting model shape and preview to 480 x 640 to keep the full FOV. Because preview will downscale the image, it is recommended you train on downsampled images as well!

So, you downsampling images sounds correct. When you start the training, make sure you specify the 480 x 640 shape. Not sure if YoloV7 supports that. If not, either using 480 or 640 square image during training should be fine. At export time (on tools.luxonis.com) make sure you specify "640 480" into the shape field. This will export your model for 480 x 640. Note: You wrote shape in HxW so I kept the same format for clarity here, we expect "W H" in tools.luxonis.com.

SeanWorcester

Hello @jakaskerl and @Matija

I am still getting errors

I created two new models one at 640x480 and another at 640x640.

I resized my training images to the model size before running train.py

for the 640x640 I used this command
!python train.py --epochs 300 --workers 8 --device 0 --batch-size 32 --data /content/yolov7/data/custom.yaml --img 640 640 --cfg cfg/training/custom_yolov7-tiny.yaml --weights 'yolov7-tiny.pt' --hyp data/hyp.scratch.tiny.yaml

to create the blob file I did a Yolov7(detection only) at RCS2 and made the input image 640 640.

For the 640x480 I used this command
!python train.py --epochs 300 --workers 8 --device 0 --batch-size 32 --data /content/yolov7/data/custom.yaml --img 640 480 --cfg cfg/training/custom_yolov7-tiny.yaml --weights 'yolov7-tiny.pt' --hyp data/hyp.scratch.tiny.yaml

When I create the this blob file I did a Yolov7(detection only) at RCS2 and made the input image 640 480

I copied both of my new blob files into depthai-python/examples/models and nnBlobPath variable in Spatial_tiny_yolo.py

for instance for my 640x640

nnBlobPath = str((Path(file).parent / Path('../models/yolo-v7-tiny tf_640x640_openvino_2022.1_6shave.blob')).resolve().absolute())

then I sent the preview size to camRgb.setPreviewSize(640, 640)

when I ran Spatial_tiny_yolo.py with either model I get the same errors

[18443010D1F3660E00] [1.2.1] [55.890] [SpatialDetectionNetwork(1)] [error] Mask is not defined for output layer with width '20'. Define at pipeline build time using: 'setAnchorMasks' for 'side20'.

[18443010D1F3660E00] [1.2.1] [56.000] [SpatialDetectionNetwork(1)] [error] Mask is not defined for output layer with width '80'. Define at pipeline build time using: 'setAnchorMasks' for 'side80'.

[18443010D1F3660E00] [1.2.1] [56.000] [SpatialDetectionNetwork(1)] [error] Mask is not defined for output layer with width '40'. Define at pipeline build time using: 'setAnchorMasks' for 'side40'

but this time the RGB and the depth outputs are show, but I see no object detection nor Spatial data.

thank you,