OAK-D: Encode frames with custom drawings (HoughCircles)

meaton

Hi,
I followed “rgb_mobilenet_4k.py” to understand how to encode video by piping the camera RGB frame to the video encoder. I would like to add an extra step in between the camera RGB / video encoder by modifying the output of the camera RGB frame with circle drawings discovered by opencv’s HoughCircles function and then send the modified frame to the video encoder (Cam RGB -> Hough Circle Algo -> Video Encoder). Is something like this possible? Currently, I have the modified camera RGB frame with circles drawn in it, but I can’t figure how to send that frame to the video encoder.
Thanks!

erik

Hello meaton,
currently, that's unfortunatelly not possible, as you can't run opencv on the device. I believe we plan on adding (some) opencv functions to the script node, so you will be able to perform something like this. I would suggest running the opencv functions on the host side, after receiving the frames from the DepthAI device.
Thanks, Erik

meaton

Thanks for the response Erik. Currently, I am running the opencv on the host side to perform the hough circle function on the frame. So, it appears transferring the modified frame from the host side to the device's video encoder is not possible. Is that a correct statement?

erik

Hello meaton,
sorry for the inaccurate answer, what you are describing is possible - sending frame back to the device and performing video encoding. I was thinking about performing the opencv functions on the device itself. But as Brandon pointed out, you could use custom NN to perform Hough Circle Function on the device itself. And as Brandon provided one link, here's another one that does Harris corner detection (custom NN): https://github.com/kunaltyagi/pytorch_harris/
Thanks, Erik

Brandon

So I'm not immediately sure if it is possible to send imagery from the host to DepthAI to do the encoding. I think it actually is possible though - we just don't have an example for it.

I think what Erik was meaning is that there's not a way to run OpenCV directly on the DepthAI device. But I think what you are proposing should be doable without a problem.

Thoughts? And @erik can confirm if I'm not wrong when he's back online (he's in Europe so likely offline now).

Also as an aside, you may be able to use something like this to run your Hough Circle Function as a neural-inference node, run directly on the camera:
https://rahulrav.com/blog/depthai_camera.html

Thanks,
Brandon

meaton

The custom NN information you provided will help as it will allow most of computation to occur on the device side (Intel Myriad) and to pipe the custom NN output to the video encoder . This leads to another question, does depthAI’s API support linking custom NN with two inputs? For example, in “rgb_mobilenet_4k.py” line 52 ( camRgb.preview.link(nn.input)), shows piping the RGB data into NN node with one input. However, PyTorch supports the ability to create models (custom NN) with two inputs ( https://discuss.pytorch.org/t/nn-module-with-multiple-inputs/237 ). The inputs of the custom NN would be the detection results of a object detection model and the camera's RGB data.

By the way, I just want to say the team, product, documentation, and examples are amazing. I am an embedded SW engineer with zero computer vision experience and less than a week I was able to get a custom object detection model working that allowed me to annotate a video stream in real time.
Thanks!

erik

Hello meaton, thank you for the kind words 🙂! This is the exact goal that we are trying to achieve, so we are going in the right direction.
There's a way to use multiple inputs to the NN, but you need to construct the NNData object (on the host/script node) and send it to the NN node. Example code of creating and setting the NNData message.
Thanks, Erik

erik

Hello meaton, I just double checked my previous answer (from 10 days ago) about the multiple inputs to the neural network and found out I was wrong. It's already possible to use a model with multiple inputs, but you need to create the NNData message, as mentioned in my edited reply to you. Sorry for the inaccurate answer.
Erik

Chris_wei

meaton hi meaton, do I understand your original post in that you were able to run the 4K mobilenet while encoding/recording the 4K h265 stream? If so, could you share how you made that work? I have been trying and so far am able to run the 4K mobilenet solo, run mobilenet in 1080p and record the stream simultaneously, and record 4K solo, but when I try to run the mobilenet in 4K while encoding the stream I get a segmentation fault or pipeline message errors.

erik is this even possible or is it a resource limitation of the hardware

thank you

naldic

Hi erik, are there any hints you can provide on how to send the frames back to the device for encoding? The messages coming back from the encoder bitstream are all None when I try it. 😕

erik

Hello naldic,
we do have a couple of sample codes that do encoding, for example rgb encoding and encoding max limit. They both send bitstream back to the host.
Thanks, Erik

erik

Hello Chris_wei , if what's possible? The multiple inputs to the NN? It's actually supported now (mainline as well, latest release 2.14 depthai). Demo here (we will get that PR merged soon now that 2.14 just released).
Thanks, Erik

Chris_wei

hi erik, I took the original statement of "I followed “rgb_mobilenet_4k.py” to understand how to encode video by piping the camera RGB frame to the video encoder." to mean that meaton was able to encode the 4k video stream being input to the NN. I was trying to run the RGB Encoding & MobilenetSSD example, it worked in 1080p, but faults when the sensor resolution is set to "THE_4_K". My goal is to record the 4K stream while running the Mobilenet, so I though this topic may be related. I can create a new topic if that would be more appropriate

erik

Hello Chris_wei, could you share the error?

Chris_wei

erik When I run the RGB Encoding & MobilenetSSD example, change the camera resolution to "THE_4_K", and change the nn path to "'../models/mobilenet-ssd_openvino_2021.4_5shave.blob'" I get the following error (I have the example named "mobilenet.py" in my folder):
Traceback (most recent call last):
File "./mobilenet.py", line 86, in <module>
inRgb = qRgb.tryGet()
RuntimeError: Communication exception - possible device error/misconfiguration. Original message 'Couldn't read data from stream: 'rgb' (X_LINK_ERROR)'

an odd observation is that if I cover the camera before running the example, the program will usually run until I uncover the camera and it detects something