Yolo8 object detection not working with ROS2 on Oak-D-PRO W

farbod_f

Hello,

I have trained a Yolo8 model to detect just one object for testing object detection with Oak-D-PRO W using ROS2.

The model can detect and localize the object pretty well when I use it with my own Python code. But, when using ROS2 from ros2 examples package, the object is detected, but the localization is jumpy and incorrect. I would like to know what I have to change in the ROS2 code from ros2 examples package to get this working.

Note: My Yolo8 was trained on 640x640 images, while the ROS2 examples package uses Yolo4 trained on 416x416 images.

Here is the pipeline code in my Python script that works: https://www.dropbox.com/scl/fi/di1f37yvaf9cixx33oxl6/pipeline.py?rlkey=74hjc4g6ky1qdepdmv7bb1zz9&st=4cgpcykb&dl=0

Here is the complete code from ROS2 examples, also showing the pipeline. Note that I have revised it and have replaced all indications of 416x416 with 640x640. Also, number of classes is 1 for my case: https://www.dropbox.com/scl/fi/506hny7evpzvszou507v2/yolov4_spatial_publisher.cpp?rlkey=o2l6jerc38h1ulf0pfca5p58t&st=vb57b0n5&dl=0

Question 1: Is ROS2 examples code supposed to work with Yolo8?
Question 2: How should I modify that ROS2 examples code to work with my Yolo8 model trained on 640x640 images?

Any clues are appreciated.

Thanks in advance!

KlemenSkrlj

Hi @farbod_f ,
comparing the two scripts you've shared there are still some differences that could be the cause of your issues:

YOLOv8 is an anchor-less model (unlike YOLOv4 used in the ROS example). That is why you can in your ROS code comment out these 2 lines:
spatialDetectionNetwork->setAnchors({10, 14, 23, 27, 37, 58, 81, 82, 135, 169, 344, 319});
spatialDetectionNetwork->setAnchorMasks({{"side13", {3, 4, 5}}, {"side26", {1, 2, 3}}});
In the ROS pipeline you set the iou threshold to 0.5 explicitly but you don't do this in the python pipeline. Maybe try removing this or changing its value?
- Lower IoU threshold means that detections can be more overlapped
In the python code you set the number of classes to len of self.label_map but in ROS you hardcode it to 1. Not sure if length of the self.label_map is also 1.

Let me know if making the scripts more unified helps.
Klemen

farbod_f

Hello,

I tried what you suggested, but the ROS2 code still does not work.

Then, I copied all the lines in the Python pipeline code into the C++ ROS2 code, converted to C++ syntax, compiled and ran it. Still, the ROS2 code is not working. Here is the revised C++ ROS2 code: https://www.dropbox.com/scl/fi/hxxxzya1eb70kkwqnl4fu/yolov4_spatial_publisher.cpp?rlkey=ejzr6pjw4pqwg961v72efiv6k&st=rmactwgm&dl=0

Is it possible the Python code is using a different Depthai library (as in import depthai as dai) than the C++ code is?

How can I make sure that both the Python code and the C++ ROS2 code use the same version of Depthai?

I suspect this, because:
stereo.setDepthAlign(dai.CameraBoardSocket.CAM_A) # Align depth to RGB camera
works in the Python code and its C++ equivalent
stereo->setDepthAlign(dai::CameraBoardSocket::CAM_A);
does not compile, and I have to use
stereo->setDepthAlign(dai::CameraBoardSocket::RGB);
to get the C++ code to compile.

The error that I get is:
/luxonis/depthai_examples/src/yolov4_spatial_publisher.cpp:75:51: error: ‘CAM_A’ is not a member of ‘dai::CameraBoardSocket’

75 | stereo->setDepthAlign(dai::CameraBoardSocket::CAM_A);

Any help is truly appreciated.
Thank you so much in advance!

jakaskerl

farbod_f
This is some old depthai no? I believe we fixed all of this to only work with CAM_A, CAM_B.. not CAM_RGB or CAM_LEFT.

Thanks,
Jaka

KlemenSkrlj

Could you do pip show depthai in your Python environment and check the version that you are using there?
For the C++ CC: @MatevzMorato to help with this

farbod_f

KlemenSkrlj

Hi there!

Thanks for the note. I am sure my questions will sound dumb to you, but please be gentle on me! :-)
Here is what I get:

pip show depthai

Name: depthai

Version: 2.30.0.0

Summary: DepthAI Python Library

Home-page: https://github.com/luxonis/depthai-python

Author: Luxonis

Author-email: support@luxonis.com

License: MIT

Location: /home/savantz/.local/lib/python3.8/site-packages

Requires:

Required-by: depthai-sdk

But doesn't show only the vestion of the Python package?

Q1: How do I get the version the C++ package I have on my computer? I am talking about the one that is sitting in /usr/local/include/depthai. As I undestand, any C++ code that I compile uses the include files in this folder. I see a file version.hpp in /usr/local/depthai/build that mentions verion 2.17.2.

Q2: Also, I do not know how to update that C++ package that is sitting in /usr/local/include/depthai? I searched in Google, but I find some intructions that want me to load and build some package. Is that the only way to update the C++ version? I also found this link: https://docs.luxonis.com/software/depthai/manual-install/#Manual%20DepthAI%20installation-Installing%20DepthAI But that only talks about the Python installation.

Thanks so much for all the help.

jakaskerl

farbod_f
There's the version in depthaiConfigVersion.cmake inside the build directory.

farbod_f Q2: Also, I do not know how to update that C++ package that is sitting in /usr/local/include/depthai? I searched in Google, but I find some intructions that want me to load and build some package. Is that the only way to update the C++ version? I also found this link: https://docs.luxonis.com/software/depthai/manual-install/#Manual%20DepthAI%20installation-Installing%20DepthAI But that only talks about the Python installation.

You'd need to install it to update the /usr/local path. cmake --build build --target install

Thanks,
Jaka