**[RESOLVED]** Hi everyone, I'm trying to get a squared image into a NN Node, apply a Semantic Segmentation model to it, and key out the alpha into the image. It's working fine using the following: A) Image Manipulation from RGB Cam 1)Image Manipulation(IM1) on Preview RGB camera -> cropping to the center to get a square image, 2)Image Manipulation on IM1 -> resizing to the size of the model I'm using ( in [this ](https://github.com/luxonis/depthai-experiments/blob/ed603060a162824a183980a9586fd8a3df453e19/gen2-deeplabv3_person/main.py) case 256x256). B) [The example provided](https://github.com/luxonis/depthai-experiments/blob/ed603060a162824a183980a9586fd8a3df453e19/gen2-deeplabv3_person/main.py) Then using a Neural network node and a RawBuffer to process the 256x256 image. The first problem is that using 2 image manipulation makes it a little slower when using the Neural network afterwards. When removing the NN part, I still get a slight delay on the resized cv::Mat : [URL=https://i.imgur.com/PYMa4hg.gif][IMG]https://i.imgur.com/PYMa4hg.gif[/IMG][/URL] I was wondering if there was an other way to get a sqaured image to be processed by the NN without having to use an image manipulator, maybe using the preview? I'm not sure what kind of offset I would get between video and preview in thise case? Second problem is the Neural Network processing seem a little choppy. I'm doing: ```cpp // startup pipeline->setOpenVINOVersion(dai::OpenVINO::VERSION_2021_4); // Define source and output auto xin = pipeline->create(); auto xoutPreview = pipeline->create(); auto xoutRgb = pipeline->create(); auto nnOut = pipeline->create(); xoutPreview->setStreamName("previewNN"); xoutRgb->setStreamName("rgb"); xin->setStreamName("nn_in"); nnOut->setStreamName("segmentation"); auto resizeManip = pipeline->create(); resizeManip->initialConfig.setResize(previewSize.x, previewSize.y); resizeManip->initialConfig.setFrameType(dai::ImgFrame::Type::BGR888p); std::tuple resVid = camRgb->getPreviewSize(); float ratio = (float(std::get(resVid)) - float(std::get(resVid))) / float(std::get(resVid)); float ratioXMin = ratio / 2.; float ratioXMax = 1.0 - ratio / 2.; offsetCrop = { ratioXMin, ratioXMax }; auto cropManip = pipeline->create(); cropManip->initialConfig.setCropRect(ratioXMin, 0, ratioXMax, 1); cropManip->setMaxOutputFrameSize(camRgbNode->getCam()->getPreviewHeight() * camRgbNode->getCam()->getPreviewWidth() * 3); xin->setMaxDataSize(300 * 300 * 3); xin->setNumFrames(30); // Linking camRgb->preview.link(cropManip->inputImage); cropManip->out.link(resizeManip->inputImage); resizeManip->out.link(detectionNN->input); //camera to screen camRgbNode->preview.link(xoutRgb->input); detectionNN->passthrough.link(xoutPreview->input); xin->out.link(detectionNN->input); // segmantation to Screen detectionNN->out.link(nnOut->input); // Connect to device and start pipeline device = new dai::Device(*pipeline, dai::UsbSpeed::SUPER); qRgb = device->getOutputQueue("previewNN", 4, false); qNN = device->getOutputQueue("segmentation", 4, false); inDataInQueue = device->getInputQueue("nn_in"); qCam = device->getOutputQueue("rgb", 4, false); tensor = std::make_shared(); ``` then in in update: ```cpp if (qCam) { std::shared_ptr inPreview = qRgb->tryGet(); std::shared_ptr inRgb = qCam->tryGet(); if (inPreview) { cv::Mat previewFrame = inPreview->getCvFrame(); toPlanar(previewFrame, tensor->data); inDataInQueue->send(tensor); std::shared_ptr inDet = qNN->tryGet(); // then update mask std::vector vecAllLayers = inDet->getAllLayers(); if (vecAllLayers.size() > 0) { std::vector layer1 = inDet->getLayerInt32(vecAllLayers[0].name); if (layer1.size() > 0) { // update ref texture } } } } ``` Is the order of the Update correct? 1) get Preview frame to tensor->data 2) send tensor with DataInputQueue 3) get NNData from DataOutputQueue The code is on github [here](https://github.com/pierrextardif/Nap-depthai-core), using [Nap Framework](https://www.napframework.com/), but i can provide it in a one filer, :). Thanks for your help, Best, P

Hello @"pierrextardif"#p3654 ! It seems like you are first sending the square image to the host computer, then cropping it (I think) and then sending it back. This adds a lot of delay. I would suggest removing this logic, and instead use ImageManip node to resize the frame to 256x256. An example of this [is here](https://github.com/luxonis/depthai-experiments/blob/master/gen2-age-gender/main.py#L15-L30), so high-res frames (1080x1080) are sent back to the host for visualization, and imageManip resized high-res frames to 300x300 which is needed by the NN model. Thoughts? Thanks, Erik

Hi Erik, Thanks for the prompt response, and trying to guide me. Yes indeed, I removed the InputQueue. I thought I needed it to use the NN, but i guess it works without it! It worked! code: ```cpp #include "depthai/depthai.hpp" using namespace std; // from utilities of depthai-core/examples/utilities void toPlanar(cv::Mat& bgr, std::vector& data) { data.resize(bgr.cols * bgr.rows * 3); for (int y = 0; y pipeline; dai::Device* device; int previewNN = 256; int main() { dai::Pipeline p; pipeline = std::make_shared(p); pipeline->setOpenVINOVersion(dai::OpenVINO::VERSION_2021_4); // Define source and output auto xin = pipeline->create(); auto xoutPreview = pipeline->create(); auto xoutRgb = pipeline->create(); auto nnOut = pipeline->create(); xoutPreview->setStreamName("previewNN"); xoutRgb->setStreamName("rgb"); xin->setStreamName("nn_in"); nnOut->setStreamName("segmentation"); shared_ptr camRgb = pipeline->create(); camRgb->setPreviewSize(1920, 1080); camRgb->setResolution(dai::ColorCameraProperties::SensorResolution::THE_1080_P); camRgb->setInterleaved(false); camRgb->setColorOrder(dai::ColorCameraProperties::ColorOrder::RGB); std::string nnPath = "C:/Users/pierr/Documents/DEV/DEPTHAI_DEV/CroppingDepthAICore/data/deeplabv3p_person_6_shaves.blob"; std::shared_ptr detectionNN = pipeline->create(); detectionNN->setBlobPath(nnPath); detectionNN->setNumInferenceThreads(2); detectionNN->input.setBlocking(false); auto resizeManip = pipeline->create(); resizeManip->initialConfig.setResize(previewNN, previewNN); resizeManip->initialConfig.setFrameType(dai::ImgFrame::Type::BGR888p); std::tuple resVid = camRgb->getPreviewSize(); float ratio = (float(std::get(resVid)) - float(std::get(resVid))) / float(std::get(resVid)); float ratioXMin = ratio / 2.; float ratioXMax = 1.0 - ratio / 2.; auto cropManip = pipeline->create(); cropManip->initialConfig.setCropRect(ratioXMin, 0, ratioXMax, 1); cropManip->setMaxOutputFrameSize(camRgb->getPreviewHeight() * camRgb->getPreviewWidth() * 3); xin->setMaxDataSize(300 * 300 * 3); xin->setNumFrames(30); // Linking camRgb->preview.link(cropManip->inputImage); cropManip->out.link(resizeManip->inputImage); //checkout out delay //cropManip->out.link(xoutPreview->input); //resizeManip->out.link(detectionNN->input); //camera to screen camRgb->preview.link(xoutRgb->input); resizeManip->out.link(xoutPreview->input); //detectionNN->passthrough.link(xoutPreview->input); //xin->out.link(detectionNN->input); // segmantation to Screen //detectionNN->out.link(nnOut->input); // Connect to device and start pipeline device = new dai::Device(*pipeline, dai::UsbSpeed::SUPER_PLUS); std::shared_ptr qRgb = nullptr; std::shared_ptr qCam = nullptr; std::shared_ptr qNN = nullptr; qRgb = device->getOutputQueue("previewNN", 4, false); qNN = device->getOutputQueue("segmentation", 4, false); qCam = device->getOutputQueue("rgb", 4, false); while (1) { std::shared_ptr inRgb = qCam->tryGet(); std::shared_ptr inPreview = qRgb->tryGet(); std::shared_ptr inDet = qNN->tryGet(); if (inRgb) { cv::Mat colorFrame = inRgb->getCvFrame(); cv::imshow("colorImg", colorFrame); } if (inPreview) { cv::Mat previewFrame = inPreview->getCvFrame(); cv::imshow("colorCropped", previewFrame); } if (inDet) { std::vector vecAllLayers = inDet->getAllLayers(); if (vecAllLayers.size() > 0) { std::vector layer1 = inDet->getLayerInt32(vecAllLayers[0].name); if (layer1.size() > 0) { //std::cout

Awesome that you got it working @"pierrextardif"#p3663 ! I just added a RESOLVED text on the top of your initial post:) Thanks, Erik

NN + Image Manipulator

pierrextardif

[RESOLVED]
Hi everyone,

I'm trying to get a squared image into a NN Node, apply a Semantic Segmentation model to it, and key out the alpha into the image.

It's working fine using the following:
A) Image Manipulation from RGB Cam
1)Image Manipulation(IM1) on Preview RGB camera -> cropping to the center to get a square image,
2)Image Manipulation on IM1 -> resizing to the size of the model I'm using ( in this case 256x256).

B) The example provided
Then using a Neural network node and a RawBuffer to process the 256x256 image.

The first problem is that using 2 image manipulation makes it a little slower when using the Neural network afterwards.
When removing the NN part, I still get a slight delay on the resized cv::Mat :

I was wondering if there was an other way to get a sqaured image to be processed by the NN without having to use an image manipulator, maybe using the preview?
I'm not sure what kind of offset I would get between video and preview in thise case?

Second problem is the Neural Network processing seem a little choppy.
I'm doing:


// startup 
 pipeline->setOpenVINOVersion(dai::OpenVINO::VERSION_2021_4);

        // Define source and output
        auto xin = pipeline->create<dai::node::XLinkIn>();
        auto xoutPreview = pipeline->create<dai::node::XLinkOut>();
        auto xoutRgb = pipeline->create<dai::node::XLinkOut>();
        auto nnOut = pipeline->create<dai::node::XLinkOut>();

        xoutPreview->setStreamName("previewNN");
        xoutRgb->setStreamName("rgb");
        xin->setStreamName("nn_in");
        nnOut->setStreamName("segmentation");



        auto resizeManip = pipeline->create<dai::node::ImageManip>();
        resizeManip->initialConfig.setResize(previewSize.x, previewSize.y);
        resizeManip->initialConfig.setFrameType(dai::ImgFrame::Type::BGR888p);


        std::tuple <int, int> resVid = camRgb->getPreviewSize();
        float ratio = (float(std::get<0>(resVid)) - float(std::get<1>(resVid))) / float(std::get<0>(resVid));


        float ratioXMin = ratio / 2.;
        float ratioXMax = 1.0 - ratio / 2.;
        offsetCrop = { ratioXMin, ratioXMax };
        auto cropManip = pipeline->create<dai::node::ImageManip>();
        cropManip->initialConfig.setCropRect(ratioXMin, 0, ratioXMax, 1);
        cropManip->setMaxOutputFrameSize(camRgbNode->getCam()->getPreviewHeight() * camRgbNode->getCam()->getPreviewWidth() * 3);


        xin->setMaxDataSize(300 * 300 * 3);
        xin->setNumFrames(30);


        // Linking
        camRgb->preview.link(cropManip->inputImage);
        cropManip->out.link(resizeManip->inputImage);
        resizeManip->out.link(detectionNN->input);


        //camera to screen
        camRgbNode->preview.link(xoutRgb->input);
        detectionNN->passthrough.link(xoutPreview->input);

        xin->out.link(detectionNN->input);

        // segmantation to Screen
        detectionNN->out.link(nnOut->input);

        // Connect to device and start pipeline
        device = new dai::Device(*pipeline, dai::UsbSpeed::SUPER);


        qRgb = device->getOutputQueue("previewNN", 4, false);
        qNN = device->getOutputQueue("segmentation", 4, false);
        inDataInQueue = device->getInputQueue("nn_in");

        qCam = device->getOutputQueue("rgb", 4, false);

        tensor = std::make_shared<dai::RawBuffer>();

then in in update:


if (qCam) {


            std::shared_ptr<dai::ImgFrame> inPreview = qRgb->tryGet<dai::ImgFrame>();
            std::shared_ptr<dai::ImgFrame> inRgb = qCam->tryGet<dai::ImgFrame>();


                if (inPreview) {
                    cv::Mat previewFrame = inPreview->getCvFrame();
                    toPlanar(previewFrame, tensor->data);
                    inDataInQueue->send(tensor);

                    std::shared_ptr<dai::NNData> inDet = qNN->tryGet<dai::NNData>();
                    // then update mask
                     std::vector<dai::TensorInfo> vecAllLayers = inDet->getAllLayers();

                    if (vecAllLayers.size() > 0) {

                        std::vector<std::int32_t> layer1 = inDet->getLayerInt32(vecAllLayers[0].name);
                        if (layer1.size() > 0) {
                           // update ref texture
                        }
                    }
                }
    }

Is the order of the Update correct?
1) get Preview frame to tensor->data
2) send tensor with DataInputQueue
3) get NNData from DataOutputQueue

The code is on github here, using Nap Framework, but i can provide it in a one filer, 🙂.

Thanks for your help,

Best,
P

erik

Hello pierrextardif !
It seems like you are first sending the square image to the host computer, then cropping it (I think) and then sending it back. This adds a lot of delay. I would suggest removing this logic, and instead use ImageManip node to resize the frame to 256x256. An example of this is here, so high-res frames (1080x1080) are sent back to the host for visualization, and imageManip resized high-res frames to 300x300 which is needed by the NN model. Thoughts?
Thanks, Erik

pierrextardif

Hi Erik,

Thanks for the prompt response, and trying to guide me.

Yes indeed, I removed the InputQueue. I thought I needed it to use the NN, but i guess it works without it!

It worked!

code:

#include "depthai/depthai.hpp"

using namespace std;


// from utilities of depthai-core/examples/utilities
void toPlanar(cv::Mat& bgr, std::vector<std::uint8_t>& data) {

    data.resize(bgr.cols * bgr.rows * 3);
    for (int y = 0; y < bgr.rows; y++) {
        for (int x = 0; x < bgr.cols; x++) {
            auto p = bgr.at<cv::Vec3b>(y, x);
            data[x + y * bgr.cols + 0 * bgr.rows * bgr.cols] = p[0];
            data[x + y * bgr.cols + 1 * bgr.rows * bgr.cols] = p[1];
            data[x + y * bgr.cols + 2 * bgr.rows * bgr.cols] = p[2];
        }
    }
}


std::shared_ptr < dai::Pipeline > pipeline;
dai::Device* device;
int previewNN = 256;

int main()
{
    dai::Pipeline p;
    pipeline = std::make_shared<dai::Pipeline>(p);
    pipeline->setOpenVINOVersion(dai::OpenVINO::VERSION_2021_4);

    // Define source and output
    auto xin = pipeline->create<dai::node::XLinkIn>();
    auto xoutPreview = pipeline->create<dai::node::XLinkOut>();
    auto xoutRgb = pipeline->create<dai::node::XLinkOut>();
    auto nnOut = pipeline->create<dai::node::XLinkOut>();

    xoutPreview->setStreamName("previewNN");
    xoutRgb->setStreamName("rgb");
    xin->setStreamName("nn_in");
    nnOut->setStreamName("segmentation");

    shared_ptr<dai::node::ColorCamera> camRgb = pipeline->create<dai::node::ColorCamera>();
    camRgb->setPreviewSize(1920, 1080);
    camRgb->setResolution(dai::ColorCameraProperties::SensorResolution::THE_1080_P);
    camRgb->setInterleaved(false);
    camRgb->setColorOrder(dai::ColorCameraProperties::ColorOrder::RGB);


    std::string nnPath = "C:/Users/pierr/Documents/DEV/DEPTHAI_DEV/CroppingDepthAICore/data/deeplabv3p_person_6_shaves.blob";
    std::shared_ptr < dai::node::NeuralNetwork > detectionNN = pipeline->create<dai::node::NeuralNetwork>();
    detectionNN->setBlobPath(nnPath);
    detectionNN->setNumInferenceThreads(2);
    detectionNN->input.setBlocking(false);




    auto resizeManip = pipeline->create<dai::node::ImageManip>();
    resizeManip->initialConfig.setResize(previewNN, previewNN);
    resizeManip->initialConfig.setFrameType(dai::ImgFrame::Type::BGR888p);


    std::tuple <int, int> resVid = camRgb->getPreviewSize();
    float ratio = (float(std::get<0>(resVid)) - float(std::get<1>(resVid))) / float(std::get<0>(resVid));


    float ratioXMin = ratio / 2.;
    float ratioXMax = 1.0 - ratio / 2.;
    auto cropManip = pipeline->create<dai::node::ImageManip>();
    cropManip->initialConfig.setCropRect(ratioXMin, 0, ratioXMax, 1);
    cropManip->setMaxOutputFrameSize(camRgb->getPreviewHeight() * camRgb->getPreviewWidth() * 3);


    xin->setMaxDataSize(300 * 300 * 3);
    xin->setNumFrames(30);


    // Linking
    camRgb->preview.link(cropManip->inputImage);   
    cropManip->out.link(resizeManip->inputImage);
    //checkout out delay
    //cropManip->out.link(xoutPreview->input); 
    //resizeManip->out.link(detectionNN->input);


    //camera to screen
    camRgb->preview.link(xoutRgb->input);
    resizeManip->out.link(xoutPreview->input);
    //detectionNN->passthrough.link(xoutPreview->input);

    //xin->out.link(detectionNN->input);

    // segmantation to Screen
    //detectionNN->out.link(nnOut->input);

    // Connect to device and start pipeline
    device = new dai::Device(*pipeline, dai::UsbSpeed::SUPER_PLUS);


    std::shared_ptr<dai::DataOutputQueue>		qRgb = nullptr;
    std::shared_ptr<dai::DataOutputQueue>		qCam = nullptr;
    std::shared_ptr<dai::DataOutputQueue>		qNN = nullptr;


    qRgb = device->getOutputQueue("previewNN", 4, false);
    qNN = device->getOutputQueue("segmentation", 4, false);
    qCam = device->getOutputQueue("rgb", 4, false);


    while (1) {


        std::shared_ptr<dai::ImgFrame> inRgb = qCam->tryGet<dai::ImgFrame>();
        std::shared_ptr<dai::ImgFrame> inPreview = qRgb->tryGet<dai::ImgFrame>();
        std::shared_ptr<dai::NNData> inDet = qNN->tryGet<dai::NNData>();


            if (inRgb) {
                cv::Mat colorFrame = inRgb->getCvFrame();
                cv::imshow("colorImg", colorFrame);
            }

            if (inPreview) {
                cv::Mat previewFrame = inPreview->getCvFrame();
                cv::imshow("colorCropped", previewFrame);
            }

            if (inDet) {

                std::vector<dai::TensorInfo> vecAllLayers = inDet->getAllLayers();

                if (vecAllLayers.size() > 0) {

                    std::vector<std::int32_t> layer1 = inDet->getLayerInt32(vecAllLayers[0].name);
                    if (layer1.size() > 0) {
                            
                        //std::cout << "getting something " << std::endl;
                    }
                }
            }






            int key = cv::waitKey(1);
            if (key == 'q') {
                return 0;
            }

           
    }

    return 0;
}

Thanks for your help,

Can you mark it as [resolved] erik ?
Best,

erik

Awesome that you got it working pierrextardif ! I just added a RESOLVED text on the top of your initial post🙂
Thanks, Erik