machine learning depthAI Hardware

zeina

Hello gays,
I am facing a problem deployed 2 ml models on OAK-PEO device with high resolution (4k).

I need to switch the model based on the trigger however, the resolution back to 1024*1024 which change the field of view of detection.
any suggestion how I can solve that?
1) is changing the color camera node with camera node could solve the problem?

2) do I need to create a pipeline and load the selected model every time the trigger fire?
please any ideas are appreciated.

PS: I can run 3 models with res:1024 * 1024

and only one model with higher resolution.

jakaskerl

zeina
I'd suggest you look through this guide on how to optimally set the input to NN so that FOV is kept.
Letterbox and stretch are generally your best bet.

zeina do I need to create a pipeline and load the selected model every time the trigger fire?
please any ideas are appreciated.

You only need to change the pipeline flow and send frame to another model instead of the current one - guide.

Thanks,
Jaka

zeina

Thank you Jaka,
I am already using letterbox. the problem is I collected images with 4k .. trained with nn with width and height 1024*1024

so I don't have problem with input to nn.

the problem is when I need to do the inference with 4k. I only be able to load 1 model because of memory limitation. but I need to deploy 2 models to use one of the models based on trigger.

is there any way to work around the 4k inference and convert it to lower resolution but keep the same width and height?
I really appreciate if there is any example showing swap models in the same oak device.

jakaskerl

zeina is there any way to work around the 4k inference and convert it to lower resolution but keep the same width and height?

Same width and height means same resolution -> same computational requirements.

lower resolution of the model (can be done fast on yolo7 and 8)
or
only use one model at a time (use script node to send frames accordingly)

Also, what do you mean by memory limitation? Do you get any errors or just low FPS?

Thanks,
Jaka

zeina

Thanks @jakaskerl for your reply.
I am trying to use script node in order to change the model path before to send it to yolo detection node.
However, I didn't find good documentation regarding input and output to script. I f I need to listen to mqtt msg and based on that I need to forward the name of the model to next node. (nn node)
I need input node and output node before and after script node.
right now, this is my pipeline. I need to find way to forward the name of the model to nn.

jakaskerl

zeina zeina
The guide I sent above (https://docs.luxonis.com/software/depthai/examples/script_change_pipeline_flow/) should have similar logic.

You'd have to integrate MQTT receiver as well. Example here:
luxonis/depthai-experimentstree/master/gen2-poe-mqtt

zeina I need to find way to forward the name of the model to nn.

Why do you need that?

Thanks,
Jaka

zeina

jakaskerl Why do you need that?

I need to change the model on fly dynamically. loading more than one model with high resolution cause memory issue (not enough memory) to create for example 3 Yolo detection nodes. so, I am trying to use the same node however change the model path whenever I receive msg from MQTT.

jakaskerl

zeina
In order to change the model without changing the pipeline flow (using Script node), the only way is to restart the pipeline.

Model path can only be changed before booting the device and not during runtime.

Thanks,
Jaka

zeina

Thank you @jakaskerl to clarify this.
In case I accept that the pipeline will be restarting. I am still not sure how I can forward the model path from scripting node to yolo node as the yolo only accept input as a frame.
Could you have example for that?

Thanks again🙂
Zeina

jakaskerl

zeina
This will not be done through depthai device.
To change the model you would need to close the device device.close(), change the pipeline.setBlobPath() then restart the pipeline each time you receive a MQTT msg.

Thanks
Jaka

zeina

@jakaskerl Do you think close the device and open it again too many times will affect the performance for oak device? I mean could shorten the life of the device.

Thanks,
Zeina

jakaskerl

zeina
No, it should not affect the life of the device.

Thanks,
Jaka

zeina

Hello again,
I was able to switch the model after closing the device.
However, for one of the models, I got this errors and warnings.. the device still open but taking more time to build the pipeline.
could you explain what is that mean? and how I solve the error?

Thanks,
Zeina

jakaskerl

zeina
What are the model parameters / size / architecture? Do you fully close the device before rerunning it or does it happen on single runs as well?

Thanks,
Jaka

zeina

I am using tiny yolo4.
before I change the model, I call device.close() then create another device and pipeline.
I am able to change between 5 models. the error just shows for one model.

jakaskerl

zeina
Ok, this points to the model being the issue. Could you try with our preloaded yolov4 model used in luxonis/depthai-pythonblob/main/examples/Yolo/tiny_yolo.py#L23 ? Does running the model on it's own work?

If not, could you send the model (not blob but .pt or whatever was used)?

Thanks,
Jaka