Hello everyone, I am a computer vision developer working for a startup which aims to make a wall-painting robot and am utilizing a double OAK-1 Lite W camera setup for obstacle detection. As a ROS newbie I need help implementing a custom pipeline for the ROS driver. I wanted to setup the pipeline which only publishes a grayscale image to improve the performance, but am having issues when launching the camera driver saying the plugin failed to load because the loaded plugin description does not exist.

I have the depthai ros packages installed using sudo apt install ros-humble-depthai-ros

As explained in the official documentation and the pipelines section, I have created a package called "custom plugins", created a header and source file containing the class derived from the BasePipeline class and have overridden the createPipeline method and exported it using PLUGINLIB_EXPORTCLASS function, followed by creating a plugins.xml and including a plugin description for the class. Finally, I have modified the i_pipeline_type to point to my plugins::GrayscalePipeline but no luck.

Below I am including the contents of the files to reproduce the error. Keep in mind the createPipeline method is the same as the createPipeline method of the RGB pipeline type since I wanted to first make sure my plugin creation works.

depthai_grayscale_plugin.hpp

#include <memory>
#include <vector>


#include "rclcpp/rclcpp.hpp"
#include "depthai_ros_driver/dai_nodes/base_node.hpp"
#include "depthai_ros_driver/pipeline/base_pipeline.hpp"

namespace plugins
{
    class GrayscalePipeline : public depthai_ros_driver::pipeline_gen::BasePipeline
    {
    public:
        std::vector<std::unique_ptr<depthai_ros_driver::dai_nodes::BaseNode>> createPipeline(
            rclcpp::Node* node,
            std::shared_ptr<dai::Device> device,
            std::shared_ptr<dai::Pipeline> pipeline,
            const std::string& nnType) override;
    };
}

depthai_grayscale_plugin.cpp

#include <memory>

#include "depthai_ros_driver/dai_nodes/sensors/sensor_wrapper.hpp"
#include "depthai_ros_driver/utils.hpp"

#include "depthai_grayscale_plugin.hpp"

std::vector<std::unique_ptr<depthai_ros_driver::dai_nodes::BaseNode>> plugins::GrayscalePipeline::createPipeline(
    rclcpp::Node * node, std::shared_ptr<dai::Device> device, std::shared_ptr<dai::Pipeline> pipeline, const std::string & nnType)
{
    auto nType = depthai_ros_driver::utils::getValFromMap(nnType, nnTypeMap);

    std::vector<std::unique_ptr<depthai_ros_driver::dai_nodes::BaseNode>> daiNodes;

    auto gray = std::make_unique<depthai_ros_driver::dai_nodes::SensorWrapper>("rgb", node, pipeline, device, dai::CameraBoardSocket::CAM_A);

    daiNodes.push_back(std::move(gray));

    return daiNodes;
}

#include "pluginlib/class_list_macros.hpp"
PLUGINLIB_EXPORT_CLASS(plugins::GrayscalePipeline, depthai_ros_driver::pipeline_gen::BasePipeline)

CMakeLists.txt

cmake_minimum_required(VERSION 3.8)
project(custom_plugins)

if(CMAKE_COMPILER_IS_GNUCXX OR CMAKE_CXX_COMPILER_ID MATCHES "Clang")
  add_compile_options(-Wall -Wextra -Wpedantic)
endif()

# find dependencies
find_package(ament_cmake REQUIRED)
find_package(pluginlib REQUIRED)
find_package(depthai_ros_driver REQUIRED)
# uncomment the following section in order to fill in
# further dependencies manually.
# find_package(<dependency> REQUIRED)

add_library(custom_plugins include/custom_plugins/depthai_grayscale_plugin.hpp src/depthai_grayscale_plugin.cpp)

target_include_directories(custom_plugins PUBLIC  
    $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include/custom_plugins>  
    $<INSTALL_INTERFACE:include/custom_plugins>  # <prefix>/include/mylib
)

ament_target_dependencies(
  custom_plugins
  depthai_ros_driver
  pluginlib
)

pluginlib_export_plugin_description_file(depthai_ros_driver plugins.xml)

install(
  TARGETS custom_plugins
  EXPORT export_${PROJECT_NAME}
  ARCHIVE DESTINATION lib
  LIBRARY DESTINATION lib
  RUNTIME DESTINATION bin
)

ament_export_libraries(
  custom_plugins
)
ament_export_targets(
  export_${PROJECT_NAME}
)

ament_package()

plugins.xml

<library path="custom_plugins">
    <class type="plugins::GrayscalePipeline" base_class_type="depthai_ros_driver::pipeline_gen::BasePipeline">
        <description>Grayscale Pipeline.</description>
    </class>
</library>

package.xml

<?xml version="1.0"?>
<?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
<package format="3">
  <name>custom_plugins</name>
  <version>0.0.0</version>
  <description>Package containing custom plugins</description>
  <maintainer email="spalj.grigor@gmail.com">grixi1405</maintainer>
  <license>Apache 2.0</license>

  <depend>pluginlib</depend>
  <depend>depthai_ros_driver</depend>

  <buildtool_depend>ament_cmake</buildtool_depend>

  <test_depend>ament_lint_auto</test_depend>
  <test_depend>ament_lint_common</test_depend>

  <export>
    <build_type>ament_cmake</build_type>
  </export>
</package>

Let me know if there is any more information you need. Thank you in advance

Hi, at the first glance the configuration looks ok, but I would try to change the C++ namespace to be consistent with the package name (so to either have plugins or custom_plugins everywhere)

14 days later

Hi, after changing the names for the sake of consistency the error is still the same. Can you verify whether this setup works or not?

Hi, I managed to get the custom pipeline working but have had issues creating a pipeline which would publish a single channel grayscale image from a color camera sensor. Is this even possible? My hopes are to decrease the load on the camera and reduce the temperature while it is running as opposed to subscribing to the color image topic and converting it to grayscale.

    Hi Gsp1405
    You'd need an image manip node (depthai API) that would convert the Color output to mono. It is not possible to do it directly from the ColorCamera node; you would get a purple stream.

    Thanks,
    Jaka