Cutting up the image before the neural network

SebastianH

Hello everyone

I have been trying to find a good way to cut of the image from the camera in 3 squares in the center of the image cause i don't really need the sides of the image. My solution right now is:

frame = qRgb.get()

aa = frame.getCvFrame()

seg = 0

for square in squares:

image_arr = aa[square[1]:square[3], square[0]:square[2]]

img = dai.ImgFrame()

img.setData(to_planar(image_arr, (320, 320)))

img.setTimestamp(monotonic())

img.setWidth(320)

img.setHeight(320)

img.setSequenceNum(seg)

qIn.send(img)

detections = qNn.get().detections

seg += 1

The squares are the 3 middle squares like this:

For my use case, each side of the squares needs to be 320 pixels.

I just need to know if this is the best way to do it or if there are more effective ways.

Thanks for reading this, I'm sorry if this is hard to follow this is my first real problem post 🙂

jakaskerl

Hi SebastianH
Try it using ImageManip node. Something like setCropRect() could work i think.

Thanks,
Jaka

SebastianH

jakaskerl Thanks for the answer I will try this.

I have another question, do you have any recommendations for some things you could do for camera tuning for an outside environment.

SebastianH

jakaskerl

Sorry if this is much to ask but I'm not sure how to do this when I need to split the same picture into 3 different pictures can you advise further? I also need to make sure that I know which is the top middle and bottom square

Thanks,

Sebastian

jakaskerl

Hi SebastianH
Here's an example. You need to basically create 3 ImageManip nodes and feed each one with a different crop rectangle.

Thanks,
Jaka

SebastianH

jakaskerl Is there a way to make sure of the order they get fed to the neural network?

jakaskerl

Hi SebastianH
What do you mean by the order? The images are usually sent at once.

Thanks,
Jaka

SebastianH

jakaskerl If i make 3 ImageManip nodes how can i be sure which one sends the picture first?

jakaskerl

Hi SebastianH
You could:

edit the model to accept 3 inputs
run 3 separate NN nodes with the same model
use Script node to correctly channel the frames through to a single NN node

In any case you are running detection with 3 x 320x320 frames which is quite a lot for edge devices like OAK so don't expect high framerate.
I guess it would be easier to train a model to accept 320x960 input (or scaled down; would be better) and feed the frames with setPreviewSize().

Thoughts?
Jaka

SebastianH

jakaskerl

I am aware of the lower framerate but this is okay for my application.

Which of numbers 2 and 3 do you believe will be best for the performance of the device?

jakaskerl

Hi SebastianH
I believe they would be pretty similar in terms of performance.
My thinking:

Option 2 - You could run camera at a higher FPS as your models are working in parallel and will be able to process the images at the same rate (limited by device NN capabilities). It makes the pipeline easier and more readable, but running 3 NNs (especially if they are consuming a lot of resources) could hinder overall detection performance.
Option 3 - You would need to run the camera at 1/3 the FPS capable by NN node since each camera frame contains 3 inference frames. Since only one NN is running, it will run faster. I think (not sure) the speed would be the same as running 3 NN in parallel. This means the overall framerate will be mostly the same.

I have never tried doing it myself so I don't know how the device would respond. Going with 2nd option is probably better as it is simpler to code and since you don't have to worry about script node blocking.

Thanks,
Jaka

SebastianH

jakaskerl Thanks so much option 2 seams to work like a charm, but i will do more testing and get back to you with the results of the performence if you would like to know.

SebastianH

jakaskerl

As of my test the detection accuracy of the model does not seem to suffer any big impact, so I would say option two is a good option 🙂

Thanks so much for the help you have given me its greatly apriciated.