Hi Luxonis team, I am using a Luxonis OAK 4 D device and I would like to benchmark an ONNX model on the device. I have been trying to use the **modelconverter benchmark** tool and I followed the available tutorials, guidelines, and instructions. However, I keep running into different issues at different stages of the process. I have already spent a lot of time (10 days) debugging your tools and approaches, but I still do not have a reliable working flow. At this point, I think the most helpful solution would be for the Luxonis team to provide a **complete working example** that can be reproduced on an OAK 4 D device. What I need is not a general explanation, but a concrete end-to-end example starting from an ONNX file and ending with benchmarking results on the device. Could you please provide the following? 1. **The exact software stack used** * SNPE version on host device **and** on OAK 4 camera. * Python version * luxonis_ml version * modelconverter version 2. **The Python script or command used to generate the NN Archive or .DLC directly** * The script/command that takes the ONNX model and creates the required archive/package 3. **The exact command used to run benchmarking on the OAK device** * Including the expected device setup * Any required configuration files * Expected output or example benchmark results I do not mind which internal procedure or toolchain you use, as long as it is reproducible. My only goal is: > I have an ONNX file, and I want to benchmark it on the OAK device. A minimal working example from Luxonis would be very helpful because it would give users a reliable reference point. Right now, the documentation and tutorials are not enough for me to get a stable working pipeline. I was able to generate NN Archive, .dlc and run 'modelconverter convert'. Then 'modelconverter benchmark' crashed during SSH transfer which I fixed (bug in '…/site-packages/modelconverter/packages/rvc4/benchmark.py'). Now when 'modelconverter benchmark' finally initiates 'snpe-parallel-run' on the device, the command fails with 'Segmentation fault (core dumped)'. After spending several days unsuccessfully setting up the workflow, I believe I should better ask you guys. I will attach my .onnx file to this post. I sincerely hope you will help me through. Thank you, joe [upl-file uuid=92fed441-2b66-4736-a63d-77d13b095f64 size=13MB]40-exp-40-half-64704844.bin[/upl-file]

Working ONNX benchmarking example for OAK 4 using modelconverter benchmark tool

joej

Hi Luxonis team,

I am using a Luxonis OAK 4 D device and I would like to benchmark an ONNX model on the device.

I have been trying to use the modelconverter benchmark tool and I followed the available tutorials, guidelines, and instructions. However, I keep running into different issues at different stages of the process. I have already spent a lot of time (10 days) debugging your tools and approaches, but I still do not have a reliable working flow.

At this point, I think the most helpful solution would be for the Luxonis team to provide a complete working example that can be reproduced on an OAK 4 D device.

What I need is not a general explanation, but a concrete end-to-end example starting from an ONNX file and ending with benchmarking results on the device.

Could you please provide the following?

The exact software stack used
- SNPE version on host device and on OAK 4 camera.
- Python version
- luxonis_ml version
- modelconverter version
The Python script or command used to generate the NN Archive or .DLC directly
- The script/command that takes the ONNX model and creates the required archive/package
The exact command used to run benchmarking on the OAK device
- Including the expected device setup
- Any required configuration files
- Expected output or example benchmark results

I do not mind which internal procedure or toolchain you use, as long as it is reproducible. My only goal is:

I have an ONNX file, and I want to benchmark it on the OAK device.

A minimal working example from Luxonis would be very helpful because it would give users a reliable reference point. Right now, the documentation and tutorials are not enough for me to get a stable working pipeline.

I was able to generate NN Archive, .dlc and run 'modelconverter convert'. Then 'modelconverter benchmark' crashed during SSH transfer which I fixed (bug in '…/site-packages/modelconverter/packages/rvc4/benchmark.py'). Now when 'modelconverter benchmark' finally initiates 'snpe-parallel-run' on the device, the command fails with 'Segmentation fault (core dumped)'. After spending several days unsuccessfully setting up the workflow, I believe I should better ask you guys. I will attach my .onnx file to this post.

I sincerely hope you will help me through.

Thank you,
joe

40-exp-40-half-64704844.bin

13MB

Luxonis-Martin

Hello,

The easiest way how to get from an ONNX model to benchmarking results is to convert the model to an NN Archive and then run the benchmarking with the default depthai backend:

$ modelconverter convert rvc4 --to nn_archive input_model shared_with_container/model.onnx
$ modelconverter benchmark rvc4 --save --model-path shared_with_container/outputs/<...>/model.rcv4.tar.xz

No configuration files are needed, you only need to have the device connected and accessible either over ADB or SSH. We advise using the default depthai backend instead of the SNPE one (that is without using the --no-dai-benchmark flag). Ensure you have depthai installed and that the camera is visible. You can verify this by running depthai -l.

The SNPE backend can lead to segmentation faults as you experienced. This can be caused by a mismatch between the SNPE version used to convert the model and the version used to run the model.

For this reason it is also advised to convert the model using the same SNPE version you have installed on your camera. You can specify the SNPE version used for the conversion by setting the --tool-version argument (e.g. --tool-version 2.32.0) will use SNPEv2.32.0 for the conversion.

To avoid the scp issue you can use the latest modelconverter version from GitHub as mentioned in my other answer.

Below you find the benchmarking results for your model. These were generated using the latest GitHub version of modelconverter. The model was converted using SNPEv2.32.6 and then benchmarked on a device with SNPEv2.41.0. Notice the version mismatch is not an issue for depthai but running the benchmark with the --no-dai-benchmark flag causes a segmentation fault.

profile,runtime,benchmark_time (s),num_threads,num_messages,device_ip,fps,latency,power_system,power_processor,ram_used,cpu_frequency,dsp_utilization,dsp_avg_frequency,dsp_power_collapse,dsp_freq_460_80,dsp_freq_576_00,dsp_freq_787_20,dsp_freq_960_00,dsp_freq_1171_20,dsp_freq_1305_60,dsp_freq_1401_60,dsp_freq_1478_40,temp_zone92,temp_zone93,temp_zone94,temp_zone95,temp_zone96,temp_avg,cpu_utilization,idle_power_system,idle_power_processor,idle_ram_used,idle_cpu_frequency,idle_dsp_utilization,idle_dsp_avg_frequency,idle_dsp_power_collapse,idle_dsp_freq_460_80,idle_dsp_freq_576_00,idle_dsp_freq_787_20,idle_dsp_freq_960_00,idle_dsp_freq_1171_20,idle_dsp_freq_1305_60,idle_dsp_freq_1401_60,idle_dsp_freq_1478_40,idle_temp_zone92,idle_temp_zone93,idle_temp_zone94,idle_temp_zone95,idle_temp_zone96,idle_temp_avg,idle_cpu_utilization
balanced,dsp,60,2,50,,30.445991939968533,,2.7065627586206897,2.1283637894736844,1314.4172952586207,613.0285714285714,85.21371479443644,1259.7995595209484,6.33,0.04,0.0,0.0,0.0,0.02,48.25,0.04,4.29,43.084431034482755,42.87979310344827,46.89338596491228,44.41071929824562,43.82315517241379,44.20656982758621,9.510743510815711,1.2249216,0.6747072000000001,940.34140625,556.8,7.085707554008991,104.75510047846892,4.7,0.06,0.0,0.0,0.0,0.0,0.0,0.0,0.34,42.653999999999996,42.7704,43.455799999999996,43.0206,43.4652,43.0732,6.256644026222627

Let us know if you run into any more issues.

Best,

Martin

Luxonis-Martin

The SNPE backend can lead to segmentation faults as you experienced. This can be caused by a mismatch between the SNPE version used to convert the model and the version used to run the model.

Small correction; this is true but in this instance the segmentation fault was caused by a different reason and will be also fixed in the upcoming release of modelconverter.

Luxonis-Martin

modelconv==0.5.5 has just been published to PyPI. It should solve both the issues you've been experiencing (a bug in scp and the Segemntation fault on the SNPE backend).

Best,

Martin

joej

Hi guys!

Thank you for you quick feedback. Martin's suggestion to use $ modelconverter convert rvc4 --to nn_archive input_model shared_withcontainer/model.onnx did the trick. Before I was using python script with luxonis_ml Archive Generator but the generated archive was missing .dlc so benchmark eventually failed. Using your suggested approach successfully generates complete nnarchive which I was able to use for benchmarking. Thanks again guys.

Looking forward to testing out the latest tools.
Thanks again!