NN + Image Manipulator

[RESOLVED]
Hi everyone,

I'm trying to get a squared image into a NN Node, apply a Semantic Segmentation model to it, and key out the alpha into the image.

It's working fine using the following:
A) Image Manipulation from RGB Cam
1)Image Manipulation(IM1) on Preview RGB camera -> cropping to the center to get a square image,
2)Image Manipulation on IM1 -> resizing to the size of the model I'm using ( in this case 256x256).

B) The example provided
Then using a Neural network node and a RawBuffer to process the 256x256 image.

The first problem is that using 2 image manipulation makes it a little slower when using the Neural network afterwards.
When removing the NN part, I still get a slight delay on the resized cv::Mat :

I was wondering if there was an other way to get a sqaured image to be processed by the NN without having to use an image manipulator, maybe using the preview?
I'm not sure what kind of offset I would get between video and preview in thise case?

Second problem is the Neural Network processing seem a little choppy.
I'm doing:

// startup 
 pipeline->setOpenVINOVersion(dai::OpenVINO::VERSION_2021_4);

        // Define source and output
        auto xin = pipeline->create<dai::node::XLinkIn>();
        auto xoutPreview = pipeline->create<dai::node::XLinkOut>();
        auto xoutRgb = pipeline->create<dai::node::XLinkOut>();
        auto nnOut = pipeline->create<dai::node::XLinkOut>();

        xoutPreview->setStreamName("previewNN");
        xoutRgb->setStreamName("rgb");
        xin->setStreamName("nn_in");
        nnOut->setStreamName("segmentation");



        auto resizeManip = pipeline->create<dai::node::ImageManip>();
        resizeManip->initialConfig.setResize(previewSize.x, previewSize.y);
        resizeManip->initialConfig.setFrameType(dai::ImgFrame::Type::BGR888p);


        std::tuple <int, int> resVid = camRgb->getPreviewSize();
        float ratio = (float(std::get<0>(resVid)) - float(std::get<1>(resVid))) / float(std::get<0>(resVid));


        float ratioXMin = ratio / 2.;
        float ratioXMax = 1.0 - ratio / 2.;
        offsetCrop = { ratioXMin, ratioXMax };
        auto cropManip = pipeline->create<dai::node::ImageManip>();
        cropManip->initialConfig.setCropRect(ratioXMin, 0, ratioXMax, 1);
        cropManip->setMaxOutputFrameSize(camRgbNode->getCam()->getPreviewHeight() * camRgbNode->getCam()->getPreviewWidth() * 3);


        xin->setMaxDataSize(300 * 300 * 3);
        xin->setNumFrames(30);


        // Linking
        camRgb->preview.link(cropManip->inputImage);
        cropManip->out.link(resizeManip->inputImage);
        resizeManip->out.link(detectionNN->input);


        //camera to screen
        camRgbNode->preview.link(xoutRgb->input);
        detectionNN->passthrough.link(xoutPreview->input);

        xin->out.link(detectionNN->input);

        // segmantation to Screen
        detectionNN->out.link(nnOut->input);

        // Connect to device and start pipeline
        device = new dai::Device(*pipeline, dai::UsbSpeed::SUPER);


        qRgb = device->getOutputQueue("previewNN", 4, false);
        qNN = device->getOutputQueue("segmentation", 4, false);
        inDataInQueue = device->getInputQueue("nn_in");

        qCam = device->getOutputQueue("rgb", 4, false);

        tensor = std::make_shared<dai::RawBuffer>();

then in in update:

if (qCam) {


            std::shared_ptr<dai::ImgFrame> inPreview = qRgb->tryGet<dai::ImgFrame>();
            std::shared_ptr<dai::ImgFrame> inRgb = qCam->tryGet<dai::ImgFrame>();


                if (inPreview) {
                    cv::Mat previewFrame = inPreview->getCvFrame();
                    toPlanar(previewFrame, tensor->data);
                    inDataInQueue->send(tensor);

                    std::shared_ptr<dai::NNData> inDet = qNN->tryGet<dai::NNData>();
                    // then update mask
                     std::vector<dai::TensorInfo> vecAllLayers = inDet->getAllLayers();

                    if (vecAllLayers.size() > 0) {

                        std::vector<std::int32_t> layer1 = inDet->getLayerInt32(vecAllLayers[0].name);
                        if (layer1.size() > 0) {
                           // update ref texture
                        }
                    }
                }
    }

Is the order of the Update correct?
1) get Preview frame to tensor->data
2) send tensor with DataInputQueue
3) get NNData from DataOutputQueue

The code is on github here, using Nap Framework, but i can provide it in a one filer, .

Thanks for your help,

Best,
P

Hi Erik,

Thanks for the prompt response, and trying to guide me.

Yes indeed, I removed the InputQueue. I thought I needed it to use the NN, but i guess it works without it!

It worked!

code:

#include "depthai/depthai.hpp"

using namespace std;


// from utilities of depthai-core/examples/utilities
void toPlanar(cv::Mat& bgr, std::vector<std::uint8_t>& data) {

    data.resize(bgr.cols * bgr.rows * 3);
    for (int y = 0; y < bgr.rows; y++) {
        for (int x = 0; x < bgr.cols; x++) {
            auto p = bgr.at<cv::Vec3b>(y, x);
            data[x + y * bgr.cols + 0 * bgr.rows * bgr.cols] = p[0];
            data[x + y * bgr.cols + 1 * bgr.rows * bgr.cols] = p[1];
            data[x + y * bgr.cols + 2 * bgr.rows * bgr.cols] = p[2];
        }
    }
}


std::shared_ptr < dai::Pipeline > pipeline;
dai::Device* device;
int previewNN = 256;

int main()
{
    dai::Pipeline p;
    pipeline = std::make_shared<dai::Pipeline>(p);
    pipeline->setOpenVINOVersion(dai::OpenVINO::VERSION_2021_4);

    // Define source and output
    auto xin = pipeline->create<dai::node::XLinkIn>();
    auto xoutPreview = pipeline->create<dai::node::XLinkOut>();
    auto xoutRgb = pipeline->create<dai::node::XLinkOut>();
    auto nnOut = pipeline->create<dai::node::XLinkOut>();

    xoutPreview->setStreamName("previewNN");
    xoutRgb->setStreamName("rgb");
    xin->setStreamName("nn_in");
    nnOut->setStreamName("segmentation");

    shared_ptr<dai::node::ColorCamera> camRgb = pipeline->create<dai::node::ColorCamera>();
    camRgb->setPreviewSize(1920, 1080);
    camRgb->setResolution(dai::ColorCameraProperties::SensorResolution::THE_1080_P);
    camRgb->setInterleaved(false);
    camRgb->setColorOrder(dai::ColorCameraProperties::ColorOrder::RGB);


    std::string nnPath = "C:/Users/pierr/Documents/DEV/DEPTHAI_DEV/CroppingDepthAICore/data/deeplabv3p_person_6_shaves.blob";
    std::shared_ptr < dai::node::NeuralNetwork > detectionNN = pipeline->create<dai::node::NeuralNetwork>();
    detectionNN->setBlobPath(nnPath);
    detectionNN->setNumInferenceThreads(2);
    detectionNN->input.setBlocking(false);




    auto resizeManip = pipeline->create<dai::node::ImageManip>();
    resizeManip->initialConfig.setResize(previewNN, previewNN);
    resizeManip->initialConfig.setFrameType(dai::ImgFrame::Type::BGR888p);


    std::tuple <int, int> resVid = camRgb->getPreviewSize();
    float ratio = (float(std::get<0>(resVid)) - float(std::get<1>(resVid))) / float(std::get<0>(resVid));


    float ratioXMin = ratio / 2.;
    float ratioXMax = 1.0 - ratio / 2.;
    auto cropManip = pipeline->create<dai::node::ImageManip>();
    cropManip->initialConfig.setCropRect(ratioXMin, 0, ratioXMax, 1);
    cropManip->setMaxOutputFrameSize(camRgb->getPreviewHeight() * camRgb->getPreviewWidth() * 3);


    xin->setMaxDataSize(300 * 300 * 3);
    xin->setNumFrames(30);


    // Linking
    camRgb->preview.link(cropManip->inputImage);   
    cropManip->out.link(resizeManip->inputImage);
    //checkout out delay
    //cropManip->out.link(xoutPreview->input); 
    //resizeManip->out.link(detectionNN->input);


    //camera to screen
    camRgb->preview.link(xoutRgb->input);
    resizeManip->out.link(xoutPreview->input);
    //detectionNN->passthrough.link(xoutPreview->input);

    //xin->out.link(detectionNN->input);

    // segmantation to Screen
    //detectionNN->out.link(nnOut->input);

    // Connect to device and start pipeline
    device = new dai::Device(*pipeline, dai::UsbSpeed::SUPER_PLUS);


    std::shared_ptr<dai::DataOutputQueue>		qRgb = nullptr;
    std::shared_ptr<dai::DataOutputQueue>		qCam = nullptr;
    std::shared_ptr<dai::DataOutputQueue>		qNN = nullptr;


    qRgb = device->getOutputQueue("previewNN", 4, false);
    qNN = device->getOutputQueue("segmentation", 4, false);
    qCam = device->getOutputQueue("rgb", 4, false);


    while (1) {


        std::shared_ptr<dai::ImgFrame> inRgb = qCam->tryGet<dai::ImgFrame>();
        std::shared_ptr<dai::ImgFrame> inPreview = qRgb->tryGet<dai::ImgFrame>();
        std::shared_ptr<dai::NNData> inDet = qNN->tryGet<dai::NNData>();


            if (inRgb) {
                cv::Mat colorFrame = inRgb->getCvFrame();
                cv::imshow("colorImg", colorFrame);
            }

            if (inPreview) {
                cv::Mat previewFrame = inPreview->getCvFrame();
                cv::imshow("colorCropped", previewFrame);
            }

            if (inDet) {

                std::vector<dai::TensorInfo> vecAllLayers = inDet->getAllLayers();

                if (vecAllLayers.size() > 0) {

                    std::vector<std::int32_t> layer1 = inDet->getLayerInt32(vecAllLayers[0].name);
                    if (layer1.size() > 0) {
                            
                        //std::cout << "getting something " << std::endl;
                    }
                }
            }






            int key = cv::waitKey(1);
            if (key == 'q') {
                return 0;
            }

           
    }

    return 0;
}

Thanks for your help,

Can you mark it as [resolved] erik ?
Best,

P